Better way to create a new column based on values of other columns

Question

What is a better way to create the same column mentioned below:

col_new = []
for r1 in df['col_A']:
    if r1==1:
        for r2 in df['col_B']:
            if r2!='None':
                col_new.append('col_new')

df['col_new'] = col_new

My dataframe is huge (120k * 22) and running the above code is hanging the notebook. Is there a faster and more efficient way to create this column where it represents all the non-null values of col_B when col_A is 1.

Do provide us with a sample data `few rows` and your desired output? — Sai Kumar, Sep 14 '18 at 05:06

jezrael · Accepted Answer · 2018-09-14T05:13:01.923

I believe need to create boolean mask and then append value by DataFrame.loc:

mask = (df['col_A'] == 1) & (df['col_B']!='None')

#if None is not string
#mask = (df['col_A'] == 1) & (df['col_B'].notnull())
df.loc[mask, 'col_new'] = 'col_new'

Sample:

In column are strings Nones:

df = pd.DataFrame({
    'col_A': [1,1,2,1],
    'col_B': ['a','None','None','a']
})
print (df)
   col_A col_B
0      1     a
1      1  None
2      2  None
3      1     a

mask = (df['col_A'] == 1) & (df['col_B']!='None')
df.loc[mask, 'col_new'] = 'val'
print (df)
   col_A col_B col_new
0      1     a     val
1      1  None     NaN
2      2  None     NaN
3      1     a     val

In column are not strings Nones, then use Series.notna:

df = pd.DataFrame({
    'col_A': [1,1,2,1],
    'col_B': ['a',None,None,'a']
})
print (df)
   col_A col_B
0      1     a
1      1  None
2      2  None
3      1     a

mask = (df['col_A'] == 1) & (df['col_B'].notna())
#oldier pandas versions
#mask = (df['col_A'] == 1) & (df['col_B'].notnull())
df.loc[mask, 'col_new'] = 'val'
print (df)
   col_A col_B col_new
0      1     a     val
1      1  None     NaN
2      2  None     NaN
3      1     a     val

Also if want use if-else statement numpy.where is really helpfull:

df['col_new'] = np.where(mask, 'val', 'another_val')
print (df)
   col_A col_B      col_new
0      1     a          val
1      1  None  another_val
2      2  None  another_val
3      1     a          val

Thank you so much. That works perfectly and is exactly what I was looking for. — Dsh M, Sep 14 '18 at 05:23

Better way to create a new column based on values of other columns

1 Answers1