5

I have two pandas Series: ser and ovr.

ser contains objects, and ovr is a sparse Series of objects and None's. ser and ovr share the same index, and I'd like to overwrite every value of ser with its corresponding value of ovr, unless that corresponding value is None.

What's an efficient way to accomplish that?

Andy Hayden
  • 359,921
  • 101
  • 625
  • 535
Dun Peal
  • 16,679
  • 11
  • 33
  • 46

1 Answers1

7

I recommend using NaN for missing data rather than None (Note: this technique also works with None).

In [1]: s1 = pd.Series([1, np.nan, 3, 4, 5, np.nan])

In [2]: s2 = pd.Series([7, 2, 3, np.nan, np.nan])

First see that s2 values which are not NaN (or None), these are those which you want to update s1 with:

In [3]: s2[s2.notnull()]
Out[3]:
0    7
1    2
2    3
dtype: float64

And then you can update the values of s1 with these:

In [4]: s1.update(s2[s2.notnull()])

In [5]: s1
Out[5]:
0     7
1     2
2     3
3     4
4     5
5   NaN
dtype: float64
Community
  • 1
  • 1
Andy Hayden
  • 359,921
  • 101
  • 625
  • 535
  • Thanks! This works well. BTW, why do you recommend NaN over None for missing values, even when the `Series` in question is of dtype `object`? – Dun Peal Sep 25 '13 at 01:10
  • 2
    @DunPeal http://stackoverflow.com/questions/17534106/what-is-the-difference-between-nan-and-none/17534682#17534682 – Andy Hayden Sep 25 '13 at 01:28
  • @AndyHayden I really needed the answer to this question but couldn't find it for ages on Google. Can we edit the question to get rid of the sparse stuff which isn't really relevant and change "override" in the title to "replace" or "update"? – LondonRob Apr 08 '14 at 12:24
  • The title's much better now! – LondonRob Apr 08 '14 at 17:24
  • It's actually even easier than this. You don't need to get rid of the nulls in `s2`. Simply `s1.update(s2)` will work. See [the docs](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.update.html): '*If other contains NaNs the corresponding values are not updated in the original Series.*' (Perhaps this API has changed in the last 7 years!) – daviewales Jun 11 '21 at 04:35