26

I am creating a dataframe from a CSV file. I have gone through the docs, multiple SO posts, links as I have just started Pandas but didn't get it. The CSV file has multiple columns with same names say a.

So after forming dataframe and when I do df['a'] which value will it return? It does not return all values.

Also only one of the values will have a string rest will be None. How can I get that column?

halfer
  • 19,824
  • 17
  • 99
  • 186
vks
  • 67,027
  • 10
  • 91
  • 124

4 Answers4

20

the relevant parameter is mangle_dupe_cols

from the docs

mangle_dupe_cols : boolean, default True
    Duplicate columns will be specified as 'X.0'...'X.N', rather than 'X'...'X'

by default, all of your 'a' columns get named 'a.0'...'a.N' as specified above.

if you used mangle_dupe_cols=False, importing this csv would produce an error.

you can get all of your columns with 

df.filter(like='a')

demonstration

from StringIO import StringIO
import pandas as pd

txt = """a, a, a, b, c, d
1, 2, 3, 4, 5, 6
7, 8, 9, 10, 11, 12"""

df = pd.read_csv(StringIO(txt), skipinitialspace=True)
df

enter image description here

df.filter(like='a')

enter image description here

piRSquared
  • 285,575
  • 57
  • 475
  • 624
  • thanx a lot!!!! works fine..... also only one can have a value in my case rest will be `NaN`.so how can i pick the only value non nan or nan is all values are nan – vks Oct 11 '16 at 21:40
  • @vks that is a completely unrelated question. It so happens that question was recently asked. I'll find it and post here. – piRSquared Oct 11 '16 at 21:45
  • 1
    @vks This question and answers can be used to get you what you want. If it isn't specific enough for you, ask a new question. http://stackoverflow.com/q/39903090/2336654 – piRSquared Oct 11 '16 at 21:48
  • 1
    I don't see you actually use the parameter `mangle_dupe_cols`. Why am I the only one confused? :) – rjurney Feb 12 '19 at 23:18
  • @rjurney you are not the only one. He needs to use it somewhere. By default if it was true then it would've handled the duplicate columns. – abhishah901 Oct 30 '19 at 21:17
  • 1
    How can I change the column names after importing and preprocessing? – abhishah901 Oct 30 '19 at 21:18
3

I had a similar issue, not due to reading from csv, but I had multiple df columns with the same name (in my case 'id'). I solved it by taking df.columns and resetting the column names using a list.

In : df.columns
Out: 
Index(['success', 'created', 'id', 'errors', 'id'], dtype='object')

In : df.columns = ['success', 'created', 'id1', 'errors', 'id2']

In : df.columns
Out: 
Index(['success', 'created', 'id1', 'errors', 'id2'], dtype='object')

From here, I was able to call 'id1' or 'id2' to get just the column I wanted.

JDenman6
  • 329
  • 2
  • 8
2

That's what I usually do with my genes expression dataset, where the same gene name can occur more than once because of a slightly different genetic sequence of the same gene:

  1. create a list of the duplicated columns in my dataframe (refers to column names which appear more than once):
duplicated_columns_list = []
list_of_all_columns = list(df.columns)
for column in list_of_all_columns:
    if list_of_all_columns.count(column) > 1 and not column in duplicated_columns_list:
        duplicated_columns_list.append(column)
duplicated_columns_list
  1. Use the function .index() that helps me to find the first element that is duplicated on each iteration and underscore it:
for column in duplicated_columns_list:
    list_of_all_columns[list_of_all_columns.index(column)] = column + '_1'
    list_of_all_columns[list_of_all_columns.index(column)] = column + '_2'

This for loop helps me to underscore all of the duplicated columns and now every column has a distinct name.

This specific code is relevant for columns that appear exactly 2 times, but it can be modified for columns that appear even more than 2 times in your dataframe.

  1. Finally, rename your columns with the underscored elements:

df.columns = list_of_all_columns

That's it, I hope it helps :)

Ofir Shorer
  • 446
  • 1
  • 6
  • 16
2

Similarly to JDenman6 (and related to your question), I had two df columns with the same name (named 'id'). Hence, calling

df['id']

returns 2 columns. You can use

df.iloc[:,ind]

where ind corresponds to the index of the column according how they are ordered in the df. You can find the indices using:

indices = [i for i,x in enumerate(df.columns) if x == 'id']

where you replace 'id' with the name of the column you are searching for.

DavidBoja
  • 21
  • 1
  • 3