better way to drop nan rows in pandas

Question

On my own I found a way to drop nan rows from a pandas dataframe. Given a dataframe dat with column x which contains nan values,is there a more elegant way to do drop each row of dat which has a nan value in the x column?

dat = dat[np.logical_not(np.isnan(dat.x))]
dat = dat.reset_index(drop=True)

Answer 1

Use dropna:

dat.dropna()

You can pass param how to drop if all labels are nan or any of the labels are nan

dat.dropna(how='any')    #to drop if any value in the row has a nan
dat.dropna(how='all')    #to drop if all values in the row are nan

Hope that answers your question!

Edit 1: In case you want to drop rows containing nan values only from particular column(s), as suggested by J. Doe in his answer below, you can use the following:

dat.dropna(subset=[col_list])  # col_list is a list of column names to consider for nan values.

Answer 2

To expand Hitesh's answer if you want to drop rows where 'x' specifically is nan, you can use the subset parameter. His answer will drop rows where other columns have nans as well

dat.dropna(subset=['x'])

Answer 3

Just in case commands in previous answers doesn't work, Try this: dat.dropna(subset=['x'], inplace = True)

Answer 4

This answer introduces the thresh parameter which is absolutely useful in some use-cases.
Note: I added this answer because some questions have been marked as duplicates directing to this page which none of the approaches here addresses such use-cases eg; The bellow df format.
Example:
This approach addresses:

1. Dropping rows/columns with all NaN
2. Keeping rows/columns with desired number of non-NaN values (having valid data)

# Approaching rows
------------------
# Sample df
df = pd.DataFrame({'Names': ['Name1', 'Name2', 'Name3', 'Name4'],
                    'Sunday': [2, None, 3, 3],
                    'Tuesday': [0, None, 3, None],
                    'Wednesday': [None, None, 4, None],
                    'Friday': [1, None, 7, None]})
print(df)

   Names  Sunday  Tuesday  Wednesday  Friday
0  Name1     2.0      0.0        NaN     1.0
1  Name2     NaN      NaN        NaN     NaN
2  Name3     3.0      3.0        4.0     7.0
3  Name4     3.0      NaN        NaN     NaN

# Keep only the rows with at least 2 non-NA values.
df = df.dropna(thresh=2)
print(df)

   Names  Sunday  Tuesday  Wednesday  Friday
0  Name1     2.0      0.0        NaN     1.0
2  Name3     3.0      3.0        4.0     7.0
3  Name4     3.0      NaN        NaN     NaN


# Keep only the rows with at least 3 non-NA values.
df = df.dropna(thresh=3)
print(df)

   Names  Sunday  Tuesday  Wednesday  Friday
0  Name1     2.0      0.0        NaN     1.0
2  Name3     3.0      3.0        4.0     7.0

---

# Approaching columns: We need axis here to direct drop to columns
------------------------------------------------------------------
# If axis=0 or not called, drop is applied to only rows like the above examples

# original df
print(df)

   Names  Sunday  Tuesday  Wednesday  Friday
0  Name1     2.0      0.0        NaN     1.0
1  Name2     NaN      NaN        NaN     NaN
2  Name3     3.0      3.0        4.0     7.0
3  Name4     3.0      NaN        NaN     NaN

# Keep only the columns with at least 2 non-NA values.
df =df.dropna(axis=1, thresh=2)                   
print(df)

   Names  Sunday  Tuesday  Friday
0  Name1     2.0      0.0     1.0
1  Name2     NaN      NaN     NaN
2  Name3     3.0      3.0     7.0
3  Name4     3.0      NaN     NaN

# Keep only the columns with at least 3 non-NA values.
df =df.dropna(axis=1, thresh=3)                  
print(df)

   Names  Sunday
0  Name1     2.0
1  Name2     NaN
2  Name3     3.0
3  Name4     3.0

Conclusion:

1. The thresh parameter from pd.dropna() doc gives you the flexibility to decide the range of non-Na values you want to keep in a row/column.
2. The thresh parameter addresses a dataframe of the above given structure which df.dropna(how='all') does not.

Answer 5

If you want to improve the readability of the code. We can have both values of Nan and notNan by using a bool series

bool_series=pd.notnull(dat["x"])
dat_notnull=dat[bool_series]
dat_null =dat[~bool_series]

Answer 6

To remove rows based on Nan value of particular column:

d= pd.DataFrame([[2,3],[4,None]])   #creating data frame
d
Output:
    0   1
0   2   3.0
1   4   NaN

d = d[np.isfinite(d[1])]  #Select rows where value of 1st column is not nan
d

Output:
    0   1
0   2   3.0

Answer 7

dropna() is probably all you need for this, but creating a custom filter may also help or be easier to understand

import pandas as pd
import numpy as np

df = pd.DataFrame(
[[4, 7, np.nan, np.nan],
[5, np.nan, 11, 2],
[6, 9, 12, np.nan]], 
index=[1, 2, 3], 
columns=['a', 'b', 'c', 'd'])
print(f'starting matrix:\n{df}')

#create the matrix of true/false NaNs:
null_matrix = df.isnull()

#create the sum of number of NaNs
sum_null_matrix = null_matrix.T.sum().T

#create the query of the matrix
query_null = sum_null_matrix<2

#apply them to your matrix
applied_df = df[query_null]
print(f'query matrix:\n{query_null}')
print(f'applied matrix:\n{applied_df}')

and you get the result:

starting matrix:
   a    b     c    d
1  4  7.0   NaN  NaN
2  5  NaN  11.0  2.0
3  6  9.0  12.0  NaN
query matrix:
1    False
2     True
3     True
dtype: bool
applied matrix:
   a    b     c    d
2  5  NaN  11.0  2.0
3  6  9.0  12.0  NaN

more info may be available on the nan checking answer: How to check if any value is NaN in a Pandas DataFrame

edit: dropna() has a threshold variable, but it doesn't have a min variable. This answer was for when someone needed to create a 'min NaNs' or some other custom function.