3

I'm trying to replace the last row(s) of a Pandas dataframe using iloc, however I cannot get it to work. There are lots of solutions out there, but the simplest (slowest) is here:

How to do a FIFO push-operation for rows on Pandas dataframe in Python?

Why doesn't this method work in the code below ?

def append_from_dataframe(self,timeframe,new_dataframe):

    new_dataframe.reset_index(inplace=True)
    temp_dataframe = self.timeframedict.get(timeframe)
    num_rows_existing = temp_dataframe.shape[0]
    num_rows_new = new_dataframe.shape[0]
    overlap = (num_rows_existing + num_rows_new) - 500
    # slow, replace with numpy array eventually
    if overlap >= 1:
        # number of rows to shift 
        i = overlap * -1
        #shift the dataframe back in time
        temp_dataframe = temp_dataframe.shift(i)
        #self.timeframedict.get(timeframe) = self.timeframedict.get(timeframe).shift(overlap)
        #replace the last i rows with the new values
        temp_dataframe.iloc[i:] = new_dataframe
        self.timeframedict.update({timeframe:temp_dataframe})

    else:
        #TODO - see this https://stackoverflow.com/questions/10715965/add-one-row-in-a-pandas-dataframe
        self.timeframedict.update({timeframe:self.timeframedict.get(timeframe).append(new_dataframe)})

Contents of the dataframe to replace one row in the other:

ipdb> new_dataframe
       Timestamp    Open    High     Low   Close   Volume      localtime
0  1533174420000  423.43  423.44  423.43  423.44  0.73765  1533174423776

temp_dataframe.shift(i) shifts value back one, replaces the values with NaN - 

ipdb> temp_dataframe.iloc[i:] 
     Timestamp  Open  High  Low  Close  Volume  localtime
499        NaN   NaN   NaN  NaN    NaN     NaN        NaN

However temp_dataframe.iloc[i:] = new_dataframe does not replace anything.

edit: I should add that after some more playing aroundnow I can replace 1 row with: 

temp_dataframe.iloc[-1] = new_dataframe.iloc[0]

however, I cannot get the multiple row version to work

Jacksporrow
  • 239
  • 5
  • 10

1 Answers1

3
df = pd.DataFrame({'a':[1,2,3,4,5],'b':['foo','bar','foobar','foobaz','food']})

Output:

df
Out[117]: 
   a       b
0  1     foo
1  2     bar
2  3  foobar
3  4  foobaz
4  5    food

Replace last two rows(foobaz and food) with second and first rows respectively:

df.iloc[-2:]=[df.iloc[1],df.iloc[0]] 

df
Out[119]: 
   a       b
0  1     foo
1  2     bar
2  3  foobar
3  2     bar
4  1     foo

You can also do this to achieve the same result:

df.iloc[-2:]=df.iloc[1::-1].values
cosmic_inquiry
  • 2,557
  • 11
  • 23
  • Thanks for pointing me in the right direction, turns out I was trying to apply a dataframe to the row which is wrong. From your suggestion I realized that `df.iloc[1::-1].values` returns a Numpy array which works great for replacing multiple rows. – Jacksporrow Aug 02 '18 at 18:37