2

I have some code that runs on a cron (via kubernetes) for several months now.

Yesterday, part of my code didn't work that normally does:

This statement, all of a sudden, wasnt 'True' (both df_temp and df_temp4 have data in them:

if ( len(df_temp > 0) & len(df_temp4 > 0)):
    print "HERE"

however, this worked:

if ( len(df_temp > 0) and len(df_temp4 > 0)):
    print "HERE"

Was there some sort of code push that would cause this change? Since I've run this code for months, not sure what would cause this statement to fail all of a sudden.

nick
  • 1,090
  • 1
  • 11
  • 24
Derek Krantz
  • 487
  • 1
  • 6
  • 18

4 Answers4

5

The len(df_temp > 0) and len(df_temp4 > 0) probably don't do what you expect. The comparison operators with pandas DataFrames return element-wise results, that means they create a boolean DataFrame where each value indicates if the corresponding value in the DataFrame is greater than zero:

>>> import pandas as pd
>>> df = pd.DataFrame({'a': [-1,0,1], 'b': [-1,0,1]})
>>> df
   a  b
0 -1 -1
1  0  0
2  1  1
>>> df > 0
       a      b
0  False  False
1  False  False
2   True   True

So the len of df is the same as the len of df > 0:

>>> len(df)
3
>>> len(df > 0)
3

difference between "&" and "and"

They mean different things:

Since you asked specifically about pandas (assuming at least one operand is a NumPy array, pandas Series, or pandas DataFrame):

  • & also refers to the element-wise "bitwise and".
  • The element-wise "logical and" for pandas isn't and but one has to use a function, i.e. numpy.logical_and.

For more explanation you can refer to "Difference between 'and' (boolean) vs. '&' (bitwise) in python. Why difference in behavior with lists vs numpy arrays?"

not sure what would cause this statement to fail all of a sudden.

You did not provide the "fail" nor the expected behavior so unfortunately I cannot help you there.

MSeifert
  • 145,886
  • 38
  • 333
  • 352
4

They have completely different behavior.
When you use and, you are comparing boolean values, but when you use & you are element-wising logical and. I suggest you to read this complete answer to learn more.
Logic operator for boolean indexing in Pandas

import pandas as pd

dfa = pd.DataFrame([True, False])
dfb = pd.DataFrame([False, False])

print(dfa & dfb)
#    0
# 0  False
# 1  False

print(dfa and dfb)
# ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
KY Lu
  • 409
  • 7
  • 12
BobyCloud
  • 244
  • 1
  • 4
  • 11
  • your answer is not correct. If you want element-wise logical-and in pandas, please use the & binary operator performs – KY Lu Sep 29 '20 at 08:03
0

You replaced wrongly applied bit operator & with proper logical and operator.

First of all, you should use and operator because what you really want is a logical expression with logical operators.

Bitwise & operator failed when lengths had no common bits.

Try the following in Python.

True and 1 & 1
True and 1 & 2
Grzegorz Żur
  • 47,257
  • 14
  • 109
  • 105
  • 1
    more specifically, I'm confused why this statement:df_temp = df[(df.funding_date >= beginning_of_month) & (df.funding_date <= yesterday) & ( df.state != 'MO')] - needs an "&" operator but the if statement needs an "and" operator – Derek Krantz Jan 22 '19 at 20:08
0

Logically the two statements are different. The & indicate bitwise operator. The AND indicates a logical and.

Rawad Daher
  • 1
  • 1