Flatten pandas dataframe column from list to their own specific column

Question

I have a pandas dataframe with a column that contains a list of arrays with information inside. it looks like this:

id   basket                                       date
c1   [{'product_id': 'P64', 'price': 1146}]       2020-08-11                                     
c2   [{'product_id': 'P44', 'price': 1426},       2020-08-11 
      {'product_id': 'P49', 'price': 1108}]                                          
c3   [{'product_id': 'P60', 'price': 39},         2020-08-11 
      {'product_id': 'P49', 'price': 1155},                                             
      {'product_id': 'P46', 'price': 178}]

I would like to flatten the basket column so it looks like this:

id   product_id  price     date
c1   P64         1146      2020-08-11                                     
c2   P44         1426      2020-08-11
c2   P49         1108      2020-08-11
c3   P60           39      2020-08-11
c3   P49         1155      2020-08-11
c3   P46          178      2020-08-11

I can't seem to figure it out, any help would be appreciated.

score 0 · Answer 1 · answered Jan 18 '21 at 15:26

Split (explode) pandas dataframe string entry to separate rows has the explode function which is great.

def explode(df, lst_cols, fill_value='', preserve_index=False):
    # make sure `lst_cols` is list-alike
    if (lst_cols is not None
        and len(lst_cols) > 0
        and not isinstance(lst_cols, (list, tuple, np.ndarray, pd.Series))):
        lst_cols = [lst_cols]
    # all columns except `lst_cols`
    idx_cols = df.columns.difference(lst_cols)
    # calculate lengths of lists
    lens = df[lst_cols[0]].str.len()
    # preserve original index values    
    idx = np.repeat(df.index.values, lens)
    # create "exploded" DF
    res = (pd.DataFrame({
                col:np.repeat(df[col].values, lens)
                for col in idx_cols},
                index=idx)
             .assign(**{col:np.concatenate(df.loc[lens>0, col].values)
                            for col in lst_cols}))
    # append those rows that have empty lists
    if (lens == 0).any():
        # at least one list in cells is empty
        res = (res.append(df.loc[lens==0, idx_cols], sort=False)
                  .fillna(fill_value))
    # revert the original index order
    res = res.sort_index()
    # reset index if requested
    if not preserve_index:        
        res = res.reset_index(drop=True)
    return res

you would call

explode(df, ['basket'], fill_value='')

then you would have to split the key and values into separate columns, this Explode dict from Pandas column does that.

Pygirl · Accepted Answer · 2021-01-18T16:20:33.407

try:

x = [pd.DataFrame(i) for i in df['basket']]
for idx, data in enumerate(x):
    data['id']=df.iloc[idx]['id']
    data['date']=df.iloc[idx]['date']
df2 = pd.concat(x).reset_index(drop=True)

df2:

    product_id  price   id  date
0   P64         1146    c1  2020-08-11
1   P44         1426    c2  2020-08-11 
2   P49         1108    c2  2020-08-11 
3   P60         39      c3  2020-08-11
4   P49         1155    c3  2020-08-11
5   P46         178     c3  2020-08-11

score 0 · Answer 3 · answered Jan 18 '21 at 15:31

You could use:

import pandas
from pandas import json_normalize

combined = pandas.concat([json_normalize(df['basket']) for column in df])

The inline-for-loop creates a list of object for every key in your column basket. Then, with pandas.concat, each list is concatenated in a dataframe and returns it to combined. I used it to flatten MongoDb query results. Afterwards you have to add the others columns.

Flatten pandas dataframe column from list to their own specific column

3 Answers3