Vectorization & ValueError, but not from "or" and "and" operators

Question

This question and answer chain do a great job explaining how to resolve ValueErrors that come up when utilizing conditionals, e.g. "or" instead of |, and "and" instead of &. But I don't see anything in that chain that resolves the problem of a "ValueError, Truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()" for vectorization when trying to use vectorization with a function that was written to take a single number as an input.

Specifically, Map and Apply work fine in this case, but Vectorization still throws the ValueError.

Code below, and can someone share how to fix this so vectorization can be used without modifying the function (or without modifying the function too much)? Thank you!

Code:

# import numpy and pandas, create dataframe.
import numpy as np
import pandas as pd
x = range(1000)
df = pd.DataFrame(data = x, columns = ['Number']) 

# define simple function to return True or False if number passed in is prime
def is_prime(num):
    if num < 2:
        return False
    elif num == 2: 
        return True
    else: 
        for i in range(2,num):
            if num % i == 0:
                return False
    return True

# Call various ways of applying the function to the data frame
df['map prime'] = list(map(is_prime, df['Number']))
df['apply prime'] = df['Number'].apply(is_prime)

# look at dataframe
in: df.head()
out: Number     map prime   apply prime
0   0   264     False       False
1   1   907     False       False
2   2   354     True        True
3   3   583     True        True
4   4   895     False       False

# now try to vectorizing
in: df['optimize prime'] = is_prime(df['Number'])
out: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Your `if/else` expression has the same problems as `or/and` - they all require a simple True/False value. — hpaulj, Feb 10 '21 at 00:47
Is there an easy way to fix that, like replaing "or" with | or "and" with &? — BLimitless, Feb 10 '21 at 00:50
The `<2` and `==2` tests are easily replaced with masking. But the `else` has a loop on `range(2,num)`, where `num` can only be a scalar. In addition the nested `if` short circuits. Applying this to the whole Series, requires a major revisualization of the task. — hpaulj, Feb 10 '21 at 01:53

score 2 · Accepted Answer · answered Feb 09 '21 at 22:52

2

You could try out numpy's vectorize:

vis_prime = np.vectorize(is_prime)

df['optimize prime'] = vis_prime(df['Number'])

That gives you:

   Number  map prime  apply prime  optimize prime
0       0      False        False           False
1       1      False        False           False
2       2       True         True            True
3       3       True         True            True
4       4      False        False           False

answered Feb 09 '21 at 22:52

rodrigocfaria

440
4
11

1

More details here: https://numpy.org/doc/stable/reference/generated/numpy.vectorize.html – rodrigocfaria Feb 09 '21 at 22:53
I'm glad it helped :D – rodrigocfaria Feb 09 '21 at 23:18

hpaulj · Answer 2 · 2021-02-10T01:48:55.290

In [133]: timeit df['map prime'] = list(map(is_prime, df['Number']))
6.25 ms ± 65.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [134]: timeit df['apply prime'] = df['Number'].apply(is_prime)
6.18 ms ± 27.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

np.vectorize has an important disclaimer - it is not a performance tool

In [137]: timeit df['optimize prime'] = vis_prime(df['Number'])
5.83 ms ± 7.28 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

It isn't much faster than pandas own apply iterator.

And with a simple list comprehension:

In [140]: timeit [is_prime(num) for num in x]
5.65 ms ± 10 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

To get real "vectorized" speedup, we have to write code that works with a whole array, or in this case, a Series.

There's an inherent serial quality to this function. In particular

        if num % i == 0:
            return False

short circuits, without iterating through the whole range(2,num). Some prime sieves also build on previous num

Vectorization & ValueError, but not from "or" and "and" operators

2 Answers2