pandas .loc returns empty dataframe

Question

I have pandas dataframe, which looks like below.

chainage(km)  
0  
0.001  
0.002  
0.003  
0.004

while I use .loc to search for the chainage(km) it returns the empty dataframe for some chainages.

print data.loc[data['chainage(km)'] == float(0.004)]

-- Empty dataframe

print data.loc[data['chainage(km)'] == float(0.001)]

-- returns value

Any help would be much appreciated.

can you put the output of `data['chainage(km)'].dtype` & `data['chainage(km)'] - float(0.004)` in your post — Haleemur Ali, May 29 '18 at 02:49
Hi Haleemur, Yep when I subtract with 0.004, then output is not 0. it is floating point math broken issue. — user55641, May 29 '18 at 03:32

score 3 · Accepted Answer · answered May 29 '18 at 03:01

3

The problem arises due to floating point inaccuracies. This is explained in Is floating point math broken?.

In situations like this, please use np.isclose instead.

df[np.isclose(data['chainage(km)'], 0.004)]

answered May 29 '18 at 03:01

cs95

379,657
97
704
746

score 0 · Answer 2 · answered May 29 '18 at 03:02

Two possible causes:

It is possible that the column chainage(km) is an object type, and for the fifth row it stores 0.004 as a string, i.e. '0.004'. To fix that cast it as float

data = data.astype(float)

If after doing the above casting, the filtering still doesn't work for 0.004

Then, given that data['chainage(km)'] == float(0.004) doesn't return any True, while data.loc[data['chainage(km)'] == float(0.001)] returns correctly, it suggest that you are experiencing floating point errors.

Try the following:

x = float(0.004)
data[abs(data['chainage(km)'] - float(x)) < 0.0001*x]

This filters for the row replacing the equality condition with an arbitrarily sized error.

pandas .loc returns empty dataframe

2 Answers2