Error when when using `scipy.stats` function with `apply()` in pandas

Question

I am trying to calculate the percentile associated with individual entries in a dataframe (using the distribution of values in a column). I am certain that I am missing some very basic thing but cannot figure out why I am getting an error when running the following code,

from scipy.stats import percentileofscore as pctl
import pandas as pd
import numpy as np

data = np.arange(100).reshape(20,5)
df = pd.DataFrame(data)

def f(series):
    r= series.index
    return pctl(series.values, series.iloc[r])

df.apply(f)

Here is the error I get,

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-4d3ad4c6f441> in <module>
----> 1 df.apply(f)

C:\Python\Miniconda\envs\leiap\lib\site-packages\pandas\core\frame.py in apply(self, func, axis, broadcast, raw, reduce, result_type, args, **kwds)
   6012                          args=args,
   6013                          kwds=kwds)
-> 6014         return op.get_result()
   6015 
   6016     def applymap(self, func):

C:\Python\Miniconda\envs\leiap\lib\site-packages\pandas\core\apply.py in get_result(self)
    316                                       *self.args, **self.kwds)
    317 
--> 318         return super(FrameRowApply, self).get_result()
    319 
    320     def apply_broadcast(self):

C:\Python\Miniconda\envs\leiap\lib\site-packages\pandas\core\apply.py in get_result(self)
    140             return self.apply_raw()
    141 
--> 142         return self.apply_standard()
    143 
    144     def apply_empty_result(self):

C:\Python\Miniconda\envs\leiap\lib\site-packages\pandas\core\apply.py in apply_standard(self)
    246 
    247         # compute the result using the series generator
--> 248         self.apply_series_generator()
    249 
    250         # wrap results

C:\Python\Miniconda\envs\leiap\lib\site-packages\pandas\core\apply.py in apply_series_generator(self)
    275             try:
    276                 for i, v in enumerate(series_gen):
--> 277                     results[i] = self.f(v)
    278                     keys.append(v.name)
    279             except Exception as e:

<ipython-input-6-347aa35ccd44> in f(series)
      1 def f(series):
      2     r= series.index
----> 3     return pctl(series.values, series.iloc[r])

C:\Python\Miniconda\envs\leiap\lib\site-packages\scipy\stats\stats.py in percentileofscore(a, score, kind)
   1785 
   1786     """
-> 1787     if np.isnan(score):
   1788         return np.nan
   1789     a = np.asarray(a)

C:\Python\Miniconda\envs\leiap\lib\site-packages\pandas\core\generic.py in __nonzero__(self)
   1574         raise ValueError("The truth value of a {0} is ambiguous. "
   1575                          "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
-> 1576                          .format(self.__class__.__name__))
   1577 
   1578     __bool__ = __nonzero__

ValueError: ('The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().', 'occurred at index 0')

[This question is also about vectorizing `scipy.stats.percentileofscore()`, although more about the highest-performance than `apply()`](https://stackoverflow.com/questions/42076126/vectorize-percentile-value-of-column-b-of-column-a-for-groups) — smci, Oct 15 '19 at 22:18

snowneji · Answer 1 · 2019-10-15T21:57:26.380

3

The problem is scipy.stats.percentileofscore(a, score, kind='rank') takes a value not an array for the score argument.

(Was posting the NumPy solution, but it's not right) How about this:

def f(series):

    return series.apply(lambda x:pctl(series,x) )

df.apply(f)

edited Oct 15 '19 at 21:57

answered Oct 15 '19 at 21:49

snowneji

1,086
1
11
25

Does not seem to work with the dataset generated above? Shouldn't the first entry be a list/array of ranks [0,...100]? – marcos Oct 15 '19 at 21:56
According to it's documentation here: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.percentileofscore.html . If I understand correctly, the first argument is an array of scores for you to compare as the reference, and then you iteratively feeding the values using the second argument. – snowneji Oct 15 '19 at 22:00

score 1 · Accepted Answer · answered Oct 15 '19 at 22:13

The score argument should be either an int or a float as per the documentation. See https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.percentileofscore.html

series.iloc[r] is a series. You need to use a scalar value for score.

Error when when using `scipy.stats` function with `apply()` in pandas

2 Answers2