0

My program generates data in each loop of a for loop function. I append to a list and finally I try to convert to the data into a single dataframe with common index and columns aligning automatically. However, I could not achieve this. My code:

biglist = []
# output of for loop in first iteration 
x1 = pd.DataFrame({'A':[11,22],'B':[33,44]},index=['x1','y1'])
biglist.append(x1)
# output of for loop in second iteration 
x2 = pd.DataFrame({'A':[110,220],'B':[330,440]},index=['x2','y2'])
biglist.append(x2)
# Now loop is over. Convert the biglist into a dataframe
df = pd.concat(biglist,axis=1)
print(df)

Present output:
      A     B      A      B
x1  11.0  33.0    NaN    NaN
y1  22.0  44.0    NaN    NaN
x2   NaN   NaN  110.0  330.0
y2   NaN   NaN  220.0  440.0
 
Expected output:
      A     B
x1  11.0   110.0
y1  22.0   220.0
x2  33.0   330.0
y2  44.0   440.0

UPDATE: My real problem is bigger. When I do pd.concat(biglist,axis=1), the output file size is [8 rows x 192 columns]. The 8 rows are perfect but 192 columns are repeated. But, when I do pd.concat(biglist,axis=0), the output file size is [64 rows x 24columns]. The 24 columns are perfect but 64 rows are repeated. Finally, the size of dataframe I am looking for is 8 rows x 24 columns. Here is the columns list.

Mainland
  • 4,110
  • 3
  • 25
  • 56
  • You don't need to do `biglist` stuff and can just do: `df = pd.concat([x1, x2], axis=1)` – David Erickson Nov 09 '20 at 22:50
  • Does this answer your question? [Pandas: append dataframe to another df](https://stackoverflow.com/questions/39815646/pandas-append-dataframe-to-another-df) – AMC Nov 09 '20 at 23:28

2 Answers2

2

By default, pd.concat will stack dataframes vertically. Pass axis=1 to do it horizontally:

df = pd.concat(biglist, axis=1)
Marat
  • 15,215
  • 2
  • 39
  • 48
  • This worked to some extent. You know what. Now I have unique index (like x,y in above in your solution) but repeatitive columns (A,B,A,B). This created a huge NaNs. My actual dataframe is big. So, what could be wrong here. – Mainland Nov 09 '20 at 22:56
  • @Mainland do you have overlapping column names in the concatenated dataframes, and if (likely) so, what is the desired result? – Marat Nov 09 '20 at 23:03
  • @Mainland Please provide a proper [mcve] that shows this. – AMC Nov 09 '20 at 23:27
  • @AMC Now I have modified my qn with a better example to illustrate the exact problem I am facing. – Mainland Nov 10 '20 at 14:16
  • @Mainland this is not really an MRE. It seems you have eight 8x24 dataframes and you want to get a single 8x24 dataframe as a result. So, each result cell has to combine 8 input cells, how do you want to do that? – Marat Nov 10 '20 at 14:42
  • @AMC I framed a new question to explain my problem. Here is https://stackoverflow.com/questions/64774519/python-append-dataframe-generated-in-nested-loops – Mainland Nov 10 '20 at 18:18
1

You have to concatenate it along axis 1 by

 df = pd.concat(biglist, axis=1)
 print(df)

    A    B
 x  11  110
 y  22  220
Omni
  • 1,002
  • 6
  • 12
  • This worked to some extent. You know what. Now I have unique index (like x,y in above in your solution) but repeatitive columns (A,B,A,B). This created a huge NaNs. My actual dataframe is big. So, what could be wrong here. – Mainland Nov 09 '20 at 22:55