How to delete dataframe rows based on a condition

Question

Here is the dataframe (csv):

Time                 Longitude      Latitude    SSHA
11/22/2013 8:57     -123.603607     81.377536   0.348
11/22/2013 8:57     -124.017502     81.387791   0.386
11/22/2013 8:57     -124.432344     81.397611   0.383
11/22/2013 8:57     -124.848099     81.406995   0.405
11/22/2013 8:57     -125.264724     81.415942   --
...                  ...            ...         ...

I want to eliminate all rows with Longitude less than 0 and greater than 40. However, when I enter my script, it doesn't work.

import pandas as pd
import numpy
df =pd.read_csv(r"C:\\Users\\chz08006\\Documents\\Results1\\BlockIsland25Test.csv")

indexNames=df[(df['Longitude'] <= 0) & (df['Longitude']>=40)].index
df.drop(indexNames,inplace=True)
df

If I just enter

indexNames=df[(df['Longitude'] <= 0)].index
df.drop(indexNames,inplace=True)
df

it works fine. However, when I add & (df['Longitude']>=40), nothing changes to the dataframe! I don't even recieve an error.

Do you mean `|`? Otherwise a number can't be `<= 0` and `>= 40`. — Joseph Crispell, Jul 26 '19 at 16:49
You don't need `.index` and `.drop`. Simply use: `df[(df['Longitude'] <= 0) |(df['Longitude']>=40)]` — Erfan, Jul 26 '19 at 16:54
Possible duplicate of [Select rows from a DataFrame based on values in a column in pandas](https://stackoverflow.com/questions/17071871/select-rows-from-a-dataframe-based-on-values-in-a-column-in-pandas) — Erfan, Jul 26 '19 at 16:54

Karn Kumar · Answer 1 · 2019-07-26T18:34:10.727

As i understand the way you are thinking that's not gona help you until there is an elegant way of doing.

Just a Example Dataset:

>>> import pandas as pd
>>> import numpy as np
>>> df
   col1 col2
0   123    A
1   123    A
2   456    B
3   456    A
4   -12    C
5    41    D

So, if you go and test with even np.bitwise_and operator you can see the bool returning False. So, thi is not going to help...

>>> np.bitwise_and(df.col1 <= 0, df.col1 >= 40)
0    False
1    False
2    False
3    False
4    False
5    False
Name: col1, dtype: bool

>>> df[np.bitwise_and(df.col1 <= 0, df.col1 >= 40)]
Empty DataFrame
Columns: [col1, col2]
Index: []

Only solution as follows as a workaround until you see a perfect way.

>>> df.drop(df.loc[df['col1'] <=0].index, inplace=True)
>>> df.drop(df.loc[df['col1'] >= 123].index, inplace=True)
>>> df
   col1 col2
5    41    D

Hope, this will help.

score 0 · Answer 2 · answered Jul 26 '19 at 17:53

As stated by @Joseph Crispell, the primary problem here is the thatx<=0 & x>=40 will always return False. So, when indexNames is instantiated, it is empty, therefore there are no indices to drop.

As @Erfran pointed out, it may be easier to use df[(df['Longitude'] <= 0) |(df['Longitude']>=40)].

But it is unclear whether you meant to say |, or if you are looking for a value between 0 and 40 (inclusive), in which case the code would look like: df[(0 <= df['Longitude'] <= 40)]

How to delete dataframe rows based on a condition

2 Answers2