pandas replace null values for a subset of columns

Question

I have a data frame with many columns, say:

df:
name   salary  age   title
John      100   35    eng
Bill      200  NaN    adm
Lena      NaN   28    NaN
Jane      120   45    eng

I want to replace the null values in salary and age, but no in the other columns. I know I can do something like this:

u = df[['salary', 'age']]
df[['salary', 'age']] = u.fillna(-1)

But this seems terse as it involves copying. Is there a more efficient way to do this?

score 25 · Answer 1 · edited Feb 18 '21 at 12:22

25

According to Pandas documentation in 23.3

values = {'salary': -1, 'age': -1}
df.fillna(value=values, inplace=True)

edited Feb 18 '21 at 12:22

easf

25
5

answered Aug 01 '18 at 17:07

Muhammad Raihan Muhaimin

5,559
7
47
68

score 16 · Accepted Answer · edited Jun 29 '22 at 17:23

16

Try this:

subset = ['salary', 'age']
df.loc[:, subset] = df.loc[:, subset].fillna(-1)

edited Jun 29 '22 at 17:23

craymichael

4,578
1
15
24

answered May 12 '16 at 16:11

piRSquared

285,575
57
475
624

Thanks to @piRSquared's answer, here's the final answer I was looking for: df.loc[:, ['salary', 'age']].fillna(-1, inplace=True) – breezymri May 12 '16 at 16:57
19

inplace=True does not work on sub-slices: https://stackoverflow.com/questions/46377263/pandas-fillna-method-does-not-work-inplace – Cheng Jun 05 '19 at 13:50
6

This does not work anymore. However `df.loc[:, ['salary', 'age']] = df.loc[:, ['salary', 'age']].fillna(-1)` does. – Francis Smart Mar 08 '21 at 17:59
1

@FrancisSmart see Cheng's comment above yours. It refers to the same thing. – piRSquared Mar 08 '21 at 18:02
1

@piRSquared Just trying to make the answer a little more accessible. – Francis Smart Mar 09 '21 at 03:07

score 7 · Answer 3 · answered May 12 '16 at 16:30

7

It is not so beautiful, but it works:

df.salary.fillna(-1, inplace=True)
df.age.fillna(-1, inplace=True)
df
>>>    name  salary   age title
    0  John   101.0  35.0   eng
    1  Bill   200.0  -1.0   adm
    2  Lena    -1.0  28.0   NaN
    3  Jane   120.0  45.0   eng

answered May 12 '16 at 16:30

xirururu

5,028
9
35
64

1

This was the only approach that worked for me. Thanks! – Alex Apr 23 '19 at 17:10

score 5 · Answer 4 · answered May 10 '21 at 05:20

5

I was hoping fillna() had subset parameter like drop(), maybe should post request to pandas however this is the cleanest version in my opinion.

df[["salary", "age"]] = df[["salary", "age"]].fillna(-1)

answered May 10 '21 at 05:20

haneulkim

4,406
9
38
80

score 2 · Answer 5 · answered Apr 14 '21 at 21:48

2

You can do:

df = df.assign(
    salary=df.salary.fillna(-1),
    age=df.age.fillna(-1),
)

if you want to chain it with other operations.

answered Apr 14 '21 at 21:48

kris

23,024
10
70
79

pandas replace null values for a subset of columns

5 Answers5

Linked