0

I am using the pandas concat method on two dataframes that number in index from 0-338 but it raises the error "InvalidIndexError: Reindexing only valid with uniquely valued Index objects". Why is this error coming up? I assumed it was because their index columns are identical but this code below runs fine despite having identical index columns.

a=pd.DataFrame([1,2,3])

b=pd.DataFrame([4,5,6])

pd.concat([a,b])

Edit: Attached a copy of the dataframe. It's basically the TagName, DateTime, Value columns repeated 36 times for different tags. df_old is a replica of this dataframe (for now, but not in general) and I run the below code to combine them which raises the error.

df_old = pd.read_excel(r'C:\Users\XXX\Downloads\Digitalisation\mat_flow\reblend_v2.xlsx')
df_merged = pd.concat([df_old, df_merged])

df_merged

                TagName            DateTime  ...            DateTime      Value
0    S041_FIC240_06B_PV 2022-01-28 05:00:00  ... 2022-02-11 09:00:00  27.493830
1    S041_FIC240_06B_PV 2022-01-28 05:30:00  ... 2022-02-11 09:30:00  27.381027
2    S041_FIC240_06B_PV 2022-01-28 06:00:00  ... 2022-02-11 10:00:00  28.407846
3    S041_FIC240_06B_PV 2022-01-28 06:30:00  ... 2022-02-11 10:30:00  27.987730
4    S041_FIC240_06B_PV 2022-01-28 07:30:00  ... 2022-02-11 11:00:00  28.378529
..                  ...                 ...  ...                 ...        ...
333                 NaN                 NaT  ... 2022-02-19 11:30:00  28.286177
334                 NaN                 NaT  ... 2022-02-19 12:00:00  28.404278
335                 NaN                 NaT  ... 2022-02-19 12:30:00  28.207792
336                 NaN                 NaT  ... 2022-02-19 13:00:00  28.002314
337                 NaN                 NaT  ... 2022-02-19 13:30:00  29.618313

[338 rows x 108 columns]
XYZ123
  • 79
  • 7
  • If you could add some code, it would be easier to understand your problem. – Hamzah Mar 24 '22 at 11:02
  • Have you tried using `ignore_index=True` as option in the concat function? – Hans Bambel Mar 24 '22 at 11:16
  • Added a copy of the dataframe @Phoenix – XYZ123 Mar 24 '22 at 12:38
  • Thanks @HansBambel but it raises the same error still when I add that option – XYZ123 Mar 24 '22 at 12:38
  • Then maybe try adding `.reset_index()` to the second dataframe – Hans Bambel Mar 24 '22 at 13:46
  • 1
    I think the problem is the duplicate column names. –  Mar 24 '22 at 15:39
  • Do you know a better way to do it then @enke? I have attached what my code is. df_old is from an excel file and I'm trying to append df_merged to the end of the file – XYZ123 Mar 25 '22 at 08:21
  • 1
    Error is expected, because if duplciated columns like `DateTime` in `a` and one column `DateTime` in `b`, pandas dont know which duplicated column from `a` match with `b` column. so error is raised. Solution is deduplicated columns names - https://stackoverflow.com/q/40774787/2901002 – jezrael Mar 25 '22 at 08:25
  • Thanks @jezrael, was able to fix it by just changing column names – XYZ123 Mar 25 '22 at 09:37

0 Answers0