1

I am struggling to remove nans. Already spent some time searching for the solution but nothing seems to work.

Below I am attaching a sample of my code. The whole notebook can be found on my GitHub here: https://github.com/jarsonX/Temp_files/blob/main/W3-Exploratory%20Data%20Analysis(1).ipynb

import pandas as pd     
import seaborn as sns               #not used in this sample, needed for plotting later on
import matplotlib as mpl            #as above
import matplotlib.pyplot as plt     #as above
import numpy as np                  #as above

df = pd.read_csv("https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/LargeData/m2_survey_data.csv")

df.Age.describe()  #dtype float64

df['Age'].isna().value_counts()  #287 nans

df['Age'].dropna(how='any', inplace=True)  #trying to remove nans

df['Age'].isna().value_counts()  #still 287 nans

#Just for the sake of identification of rows
#I tried to print ONLY nans but could not figure out how to do it.
i = 0
for el in df.Age:
    print(i, el, type(el))
    i += 1

#The first nan is in the 67th row

What am I missing?

UPDATE:

I've managed to filter out nans:

i = 0
for el in df.Age:
    if el != el:
        print(i, el, type(el))
    i += 1
jarsonX
  • 117
  • 1
  • 8
  • You need to select the whole dataframe, not just the column with your null values. Here, you're effectively dropping the null values, just you're putting them back to the same dataframe, and pandas fills the missing values with nan again. Check this: https://stackoverflow.com/questions/13413590/how-to-drop-rows-of-pandas-dataframe-whose-value-in-a-certain-column-is-nan – Emmanuel Murairi Aug 21 '22 at 10:20

2 Answers2

1

This should work:

df = df[~df['Age'].isnull()]

The ~ operator works as a negation
To verify the nans have been removed

df['Age'].isna().value_counts()

Helena Oliveira
  • 11
  • 2
1

You can try out the following snippet, dropna when called in a Series doesn't respect the how argument, since its just a single column

df.dropna(subset=["Age"], how="any", inplace=True)

Devorein
  • 1,112
  • 2
  • 15
  • 23