pandas query None values

Question

I have a dataframe with columns of different dtypes and I need to use pandas.query to filter the columns.

Columns can include missing values: NaN, None and NaT and I need to display the rows that contain such values. Is there a way to do this in an expression passed to pandas.query? I am aware that it can be done using different methods but I need to know if it is doable through the query

For boolean columns I was able to use a workaround by stating:

df.query('col not in (True, False)')

but this won't work for other types of columns. Any help is appreciated including workarounds.

@M.Massias the OP is asking for a query statement that is compatible with `df.query` method, the issue here is how to pass a statement that will work as desired using `numexpr` — EdChum, Aug 25 '15 at 19:04

score 17 · Accepted Answer · answered Aug 25 '15 at 15:16

17

NaN is not equal to itself, so you can simply test if a column is equal to itself to filter it. This also seems to work for None although I'm not sure why, it may be getting cast to NaN at some point during the evaluation.

 df.query('col == col')

For datetimes, this works, but feels pretty hacky, there might be a better way.

df.query('col not in [@pd.NaT]')

answered Aug 25 '15 at 15:16

chrisb

49,833
8
70
70

Too bad that the second approach does not work for NaN and None! However the first works perfectly for me – architectonic Aug 26 '15 at 07:08
For me, the second one work for None. – Kharthigeyan Apr 01 '21 at 19:59

pandas query None values

1 Answers1