IndexError: index 1491188345 is out of bounds for axis 0 with size 1491089723

Question

I have a dataframe,df with 646585 rows and 3 columns which looks like :

index      inp       aco         count                
  0       2.3.6.   dp-ptp-a2f   22000
  1       2.3.12.  ft-ptp-a2f   21300
  2       2.5.9.   dp-ptp-a2f   21010
  3       0.8.0.   dp-ptp-a4f   20000
  4       2.3.6.   ft-ptp-a2f   19000               
  5       2.3.6.   ff-ptp-a2f   18500        
  ...                            ...
  ...                            ...   
  ...                            ...

I tried to pivot the dataframe using the code:

df1=df.pivot_table(values='count', index='inp', columns='aco',fill_value=0)
print(df1)

but I got

IndexError: index 1491188345 is out of bounds for axis 0 with size 1491089723

score 11 · Accepted Answer · answered Jan 28 '18 at 23:51

11

There is open Pandas issue describing this error.

AFAIK currently there is no solution/workaround provided, except reducing the data set

answered Jan 28 '18 at 23:51

MaxU - stand with Ukraine

205,989
36
386
419

Has this been resolved? Should I assume that it hasn't since the issue is still open? – Alex Metsai Feb 25 '21 at 12:11
@AlexMetsai, the current status says - "Open". I guess the pandas team would change the status if they would fix it ) – MaxU - stand with Ukraine Feb 25 '21 at 12:19
So it's not a memory issue, right? Huge dataframes cause int32 fault, no matter how large your RAM is? – Alex Metsai Feb 25 '21 at 12:32
@AlexMetsai, actually it could be caused by memory error. You can have a look here: https://github.com/pandas-dev/pandas/issues/10554 – MaxU - stand with Ukraine Feb 25 '21 at 12:39
1

I see. I was thinking of openining a question, but I don't think that I could be specific enough. Anyway, thank you for your help! – Alex Metsai Feb 25 '21 at 12:42
@MaxU Can we use PySpark instead of using Pandas? Will that be a good workaround?? – Purushothaman Srikanth Mar 29 '21 at 15:24
1

@PurushothamanSrikanth, at least we can try) Unfortunately there is no `.pivot_table()` in PySpark, so one would have to do it manually: `df.groupby(...).pivot(...)...` – MaxU - stand with Ukraine Mar 29 '21 at 17:20

IndexError: index 1491188345 is out of bounds for axis 0 with size 1491089723

1 Answers1

Linked

Related