iterrows pandas get next rows value

Question

I have a df in pandas

import pandas as pd
df = pd.DataFrame(['AA', 'BB', 'CC'], columns = ['value'])

I want to iterate over rows in df. For each row i want rows value and next rows value Something like(it does not work):

for i, row in df.iterrows():
     print row['value']
     i1, row1 = next(df.iterrows())
     print row1['value']

As a result I want

'AA'
'BB'
'BB'
'CC'
'CC'
*Wrong index error here

At this point i have mess way to solve this

for i in range(0, df.shape[0])
   print df.irow(i)['value']
   print df.irow(i+1)['value']

Is there more efficient way to solve this issue?

alisdt · Accepted Answer · 2015-06-19T14:07:17.827

33

Firstly, your "messy way" is ok, there's nothing wrong with using indices into the dataframe, and this will not be too slow. iterrows() itself isn't terribly fast.

A version of your first idea that would work would be:

row_iterator = df.iterrows()
_, last = row_iterator.next()  # take first item from row_iterator
for i, row in row_iterator:
    print(row['value'])
    print(last['value'])
    last = row

The second method could do something similar, to save one index into the dataframe:

last = df.irow(0)
for i in range(1, df.shape[0]):
    print(last)
    print(df.irow(i))
    last = df.irow(i)

When speed is critical you can always try both and time the code.

edited Jun 19 '15 at 14:07

answered Apr 18 '14 at 09:57

alisdt

509
4
6

4

I believe the 2nd line of the first option should read: `_, last = row_iterator.next()` – maxliving Feb 05 '15 at 20:14
11

For Python3 folks use either next(row_iterator) or row_iterator__next__() – Sebastian Zaba Jun 28 '18 at 16:45
2

In this example what does the `_,` do? Why can't I just do `last = next(row_iterator)` and which of the [five operations](https://stackoverflow.com/questions/5893163/what-is-the-purpose-of-the-single-underscore-variable-in-python) is it performing here? – Scott Sep 04 '20 at 06:38

score 15 · Answer 2 · answered Apr 18 '14 at 13:23

15

There is a pairwise() function example in the itertools document:

from itertools import tee, izip
def pairwise(iterable):
    "s -> (s0,s1), (s1,s2), (s2, s3), ..."
    a, b = tee(iterable)
    next(b, None)
    return izip(a, b)

import pandas as pd
df = pd.DataFrame(['AA', 'BB', 'CC'], columns = ['value'])

for (i1, row1), (i2, row2) in pairwise(df.iterrows()):
    print i1, i2, row1["value"], row2["value"]

Here is the output:

0 1 AA BB
1 2 BB CC

But, I think iter rows in a DataFrame is slow, if you can explain what's the problem you want to solve, maybe I can suggest some better method.

answered Apr 18 '14 at 13:23

HYRY

94,853
25
187
187

This is very good. I was working on a similar problem to the original question and this solved my issue perfectly. Thanks. – Eric D. Brown D.Sc. Nov 28 '16 at 15:43
1

In Python3 you no longer need to import `izip` -- the inbuilt `zip` provides the functionality [SO reference](https://stackoverflow.com/questions/32659552/izip-not-working-in-python-3-x) – tatlar Dec 13 '18 at 16:43
1

direct link to Python3 `itertools` [recipe](https://docs.python.org/3/library/itertools.html#itertools-recipes) – XoXo Mar 04 '19 at 13:20

score 10 · Answer 3 · answered Jan 08 '19 at 21:03

I would use shift() function as follows:

df['value_1'] = df.value.shift(-1)
[print(x) for x in df.T.unstack().dropna(how = 'any').values];

which produces

AA
BB
BB
CC
CC

This is how the code above works:

Step 1) Use shift function

df['value_1'] = df.value.shift(-1)
print(df)

produces

value value_1
0    AA      BB
1    BB      CC
2    CC     NaN

step 2) Transpose:

df = df.T
print(df)

produces:

          0   1    2
value    AA  BB   CC
value_1  BB  CC  NaN

Step 3) Unstack:

df = df.unstack()
print(df)

produces:

0  value       AA
   value_1     BB
1  value       BB
   value_1     CC
2  value       CC
   value_1    NaN
dtype: object

Step 4) Drop NaN values

df = df.dropna(how = 'any')
print(df)

produces:

0  value      AA
   value_1    BB
1  value      BB
   value_1    CC
2  value      CC
dtype: object

Step 5) Return a Numpy representation of the DataFrame, and print value by value:

df = df.values
[print(x) for x in df];

produces:

AA
BB
BB
CC
CC

This one worked very well thank you. This line ```df['value_1'] = df.value.shift(-1)``` was all that I needed to solve my similar requirement. Thank You! — Roan, Nov 15 '19 at 04:13

score 3 · Answer 4 · edited May 05 '15 at 20:09

This can be solved also by izipping the dataframe (iterator) with an offset version of itself.

Of course the indexing error cannot be reproduced this way.

Check this out

import pandas as pd
from itertools import izip

df = pd.DataFrame(['AA', 'BB', 'CC'], columns = ['value'])   

for id1, id2 in izip(df.iterrows(),df.ix[1:].iterrows()):
    print id1[1]['value']
    print id2[1]['value']

which gives

AA
BB
BB
CC

score 2 · Answer 5 · answered May 30 '19 at 11:35

a combination of answers gave me a very fast running time. using the shift method to create new column of next row values, then using the row_iterator function as @alisdt did, but here i changed it from iterrows to itertuples which is 100 times faster.

my script is for iterating dataframe of duplications in different length and add one second for each duplication so they all be unique.

# create new column with shifted values from the departure time column
df['next_column_value'] = df['column_value'].shift(1)
# create row iterator that can 'save' the next row without running for loop
row_iterator = df.itertuples()
# jump to the next row using the row iterator
last = next(row_iterator)
# because pandas does not support items alteration i need to save it as an object
t = last[your_column_num]
# run and update the time duplications with one more second each
for row in row_iterator:
    if row.column_value == row.next_column_value:
         t = t + add_sec
         df_result.at[row.Index, 'column_name'] = t
    else:
         # here i resetting the 'last' and 't' values
         last = row
         t = last[your_column_num]

Hope it will help.

iterrows pandas get next rows value

5 Answers5

Linked