python pandas: split comma-separated column into new columns - one per value

Question

I have a dataframe like this:

data = np.array([["userA","event2, event3"],
            ['userB',"event3, event4"],
            ['userC',"event2"]])

data = pd.DataFrame(data)

        0         1
0   userA   "event2, event3"
1   userB   "event3, event4"
2   userC   "event2"

now I would like to get a dataframe like this:

       0    event2      event3      event4
0   userA     1           1
1   userB                 1           1
2   userC     1

can anybody help please?

Answer 1

It seems you need get_dummies with replace 0 to empty strings:

df = data[[0]].join(data[1].str.get_dummies(', ').replace(0, ''))
print (df)
       0 event2 event3 event4
0  userA      1      1       
1  userB             1      1
2  userC      1              

Detail:

print (data[1].str.get_dummies(', '))
   event2  event3  event4
0       1       1       0
1       0       1       1
2       1       0       0

Answer 2

If you have a lot of features (words), then it makes sense to use sparse matrices in order to use memory much more efficiently:

In [120]: from sklearn.feature_extraction.text import CountVectorizer

In [121]: cvect = CountVectorizer()

In [122]: data = data.join(pd.SparseDataFrame(cvect.fit_transform(data.pop(1)),
                                              data.index,
                                              cvect.get_feature_names(),
                                              default_fill_value=0))

In [123]: data
Out[123]:
       0  event2  event3  event4
0  userA       1       1       0
1  userB       0       1       1
2  userC       1       0       0

In [124]: data.memory_usage()
Out[124]:
Index     80
0         24
event2    16
event3    16
event4     8
dtype: int64