pandas - append new row with a different number of columns

Question

I have a dataframe (df_original) that looks like this:

Year | Thing1 | Thing2
2010 |  748   |   461
2011 |  246   |   493
2012 |  394   |   731

I need to occassionally append new data. Sometimes, I get data that will have the same number of headers. Sometimes, I won't. I also cannot count on the order of the headers being the same.

Appending something like the following would be fine:

Year | Thing1 | Thing2
2013 |  561   |  197

However, I could also receive this instead, a new thing (df_new):

Year | Thing1 | Thing2 | Thing3
2013 |  561   |   197  |   369

Edit: I could also receive this dataframe to append, will concat account for headers in a different order?

Year | Thing1 | Thing3 | Thing2
2013 |  561   |   369  |   197

The desired output is the following:

Year | Thing1 | Thing2 | Thing3
2010 |  748   |  461   |   0
2011 |  246   |  493   |   0
2012 |  394   |  731   |   0
2013 |  561   |  197   |  369

How can I achieve this result?

I thought pd.concat([df_original, df_new], join = 'outer', axis = 1, sort = False) might do it because of this post but it puts the new data to the right of the original data in df_original.

Change axis=1 to axis=0 – Michael Gardner Oct 09 '19 at 00:32 — Michael Gardner, Oct 09 '19 at 00:32

score 7 · Accepted Answer · answered Oct 09 '19 at 00:33

Simply chain pd.concat with fillna(0):

pd.concat([df, df_new], sort=False).fillna(0)

Output

   Year  Thing1  Thing2  Thing3
0  2010     748     461     0.0
1  2011     246     493     0.0
2  2012     394     731     0.0
0  2013     561     197   369.0

If you want to reset the index, use ignore_index:

pd.concat([df, df_new], sort=False, ignore_index=True).fillna(0)

output

   Year  Thing1  Thing2  Thing3
0  2010     748     461     0.0
1  2011     246     493     0.0
2  2012     394     731     0.0
3  2013     561     197   369.0

score 1 · Answer 2 · answered Oct 09 '19 at 00:35

A concat should do the trick. But you want axis 0, which is default, to add rows rather than columns, and to ignore the index values.

df = pd.concat([df_original, df_new], sort=False, ignore_index=True)

This will put Nan's instead of 0's as you have above, but you can replace those if you want.

pandas - append new row with a different number of columns

2 Answers2