How to check the presence of np.NaN and/or None in a list

Question

None is considered as false condition. But np.NaN is considered as True when converted as bool.

any([np.nan, np.NaN, None])
#True

I understand, it can be caught using np.isnan(). If my data has combination of np.NaN, None, np.nan, then how can I find it if any of these is present.

Is there any simpler way to find either of them?

EDIT:

Expected Output:

func([np.nan, np.NaN, None, 2])
#True
func([np.nan, np.NaN, None,''])
#False

related and probable dupe: https://stackoverflow.com/questions/15686318/why-do-not-a-number-values-equal-true-when-cast-as-boolean-in-python-numpy — EdChum, Jan 25 '19 at 16:40
what is the best way to know to identify `[np.nan, np.NaN, None]` as False — Venkatachalam, Jan 25 '19 at 16:41
https://stackoverflow.com/q/15686318/4799172 but for the last part, it's a dupe — roganjosh, Jan 25 '19 at 16:41
sorry what are you trying to test here? If there is `NaN` or `None`? — EdChum, Jan 25 '19 at 16:42
It should be `False`, if all the values are belonging to `[np.nan, np.NaN, None]` — Venkatachalam, Jan 25 '19 at 16:44
You can build a mask using `np.isin(your_array, [np.nan, np.NaN, None])` — Tarifazo, Jan 25 '19 at 16:48

score 3 · Answer 1 · answered Jan 25 '19 at 16:43

3

You can define your own custom function for checking whether a number is within your valid input set, for instance:

def isvalid(number):
    if number is None or np.isnan(number):
        return False
    else:
        return True

any([isvalid(x) for x in data])

edit: I realize you probably want to check if any of them are NOT valid, you can simply negate the isvalid result to achieve that.

answered Jan 25 '19 at 16:43

Cory Nezin

1,551
10
22

return the boolean, rather than True/False, e.g. `return not (number is None or np.isnan(number))` – Chris_Rands Jan 25 '19 at 16:49

score 2 · Answer 2 · answered Jan 25 '19 at 16:48

2

Another method is to convert any nan to valid numbers, in this case 0.0 :

In[91]:
any([np.nan_to_num(x) for x in [np.nan, np.NaN, None]])

Out[91]: False

np.nan_to_num will convert the NaN to 0.0:

In[92]:

[np.nan_to_num(x) for x in [np.nan, np.NaN, None]]
Out[92]: [0.0, 0.0, None]

EDIT

this will also handle your updated sample data:

In[93]:

any([np.nan_to_num(x) for x in [np.nan, np.NaN, None,'']])
Out[93]: False

answered Jan 25 '19 at 16:48

EdChum

376,765
198
813
562

l liked the idea of using `nan_to_num` to handle None,empty string and Nan. Thanks – Venkatachalam Jan 25 '19 at 16:55

Scott Boston · Accepted Answer · 2019-01-25T21:36:10.970

Here's another trick:

arr = [np.nan, None, np.nan]

[i == i if i is not None else False  for i in arr]

Output:

[False, False, False]

Note: np.nan == np.nan returns False

Hence,

arr = [np.nan, None, np.nan, '']
any([i == i if i is not None and i != '' else False  for i in arr])

Output:

False

And,

arr = [np.nan, None, np.nan, 2]

any([i == i if i is not None and i != '' else False  for i in arr])

Output:

True

Timings:

%timeit any([np.nan_to_num(x) for x in arr])

241 µs ± 22 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit any([isvalid(x) for x in arr])

20.5 µs ± 1.95 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit any([i == i if i is not None and i != '' else False  for i in arr])

2.44 µs ± 137 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Appreciate your help. Thanks – Venkatachalam Jan 26 '19 at 02:11 — Venkatachalam, Jan 26 '19 at 02:11

score 1 · Answer 4 · answered Jan 22 '21 at 20:44

Following method considers None, various NaNs and even nan as string:

>>> import numpy as np
>>> any([~np.isnan(np.float_(x)) for x in [np.nan, np.NaN, None, 'NaN', 'nan']])
False

Pandas has a notna function that works for None and various NaNs (but not NaN as a string).

>>> import pandas as pd
>>> any([pd.notna(x) for x in [np.nan, np.NaN, None, pd.NA, pd.NaT]])
False

Neither of these methods work though for the empty string ''.

How to check the presence of np.NaN and/or None in a list

4 Answers4

EDIT