Merge two dataframes on a column of lists

Question

I have a two dataframes.

df = pd.DataFrame([[1,2,3,[4,5]],[6,7,8,[9,10]]], columns=['a','b','c','d'])

df2 = pd.DataFrame([[4,'abc'],[5,'ef'], [10,'g'], [12,'hijk']], columns=['a_2','b_2'])

In [151]: df
Out[151]: 
   a  b  c        d
0  1  2  3   [4, 5]
1  6  7  8  [9, 10]

In [152]: df2
Out[152]: 
   a_2   b_2
0    4   abc
1    5    ef
2   10     g
3   12  hijk

I want to merge the two based on column 'd' of df and get the following output-

df3 = pd.DataFrame([[1,2,3,[4,5],['abc','ef']],[6,7,8,[9,10],['g']]], columns=['a','b','c','d','b_2'])


In [153]: df3
Out[153]: 
   a  b  c        d        b_2
0  1  2  3   [4, 5]  [abc, ef]
1  6  7  8  [9, 10]        [g]

I did try with 'merge' but I am not getting the required results.

score 2 · Accepted Answer · answered Jun 13 '19 at 04:14

This isn't exactly a merge problem, but I'd do this using a list comprehension that calls Series.get:

s = df2.set_index('a_2')['b_2']  # mapping to use
[[s.get(y) for y in x if y in s] for x in df['d']]
# [['abc', 'ef'], ['g']]

# df['b_2'] = [[s.get(y) for y in x if y in s] for x in df['d']]
df3 = df.assign(b_2=[[s.get(y) for y in x if y in s] for x in df['d']])
df3

   a  b  c        d        b_2
0  1  2  3   [4, 5]  [abc, ef]
1  6  7  8  [9, 10]        [g]

Evidence suggests that list comprehensions should be reasonably fast given the non-vectorizable nature of your problem.

Merge two dataframes on a column of lists

1 Answers1