merging a list of nested dictionaries

Question

I have two list of dictionaries as shown in example below

list1=[
        {
            "pdpData":{
                "a":1,
                "b":2
            }
        }
    ]

list2=[
    {
        "pdpData":{
            "a":1,
            "c":3
        }
    },
    {
        "pdpData":{
            "a":2,
            "b":3
        }
    }
]

I want the result as shown in the format below

list3=[
{
    "pdpData":{
        "a":1,
        "b":2,
        "c":3
    }
},
{
    "pdpData":{
        "a":2,
        "b":3
    }
}
]

The size of list1 and list2 could be in 10000's. List3 would be the union of list1 and list2. What could be the best pythonic solutions to solve this problem.

try this https://stackoverflow.com/questions/19561707/python-merge-two-lists-of-dictionaries — Van Peer, Oct 13 '17 at 08:41
How do you "union" the lists when they have "10000s" of elements? Do you want to union each pairs of dicts, i.e. the product, resulting in 10,000²=100,000,000 dicts in list 3? Or do you always merge all the dicts in list1 into all the dicts in list2, resulting in only 10,000 dicts in list 3? You should provide more explanation and/or examples. — tobias_k, Oct 13 '17 at 08:54

score 1 · Accepted Answer · answered Oct 13 '17 at 08:44

You didn't write any code, so I won't write a complete solution. You'll need zip_longest and dict merging.

from itertools import zip_longest

list1=[
        {
            "pdpData":{
                "a":1,
                "b":2
            }
        }
    ]

list2=[
    {
        "pdpData":{
            "a":1,
            "c":3
        }
    },
    {
        "pdpData":{
            "a":2,
            "b":3
        }
    }
]


for d1, d2 in zip_longest(list1, list2):
    dd1 = d1.get("pdpData", {}) if d1 else {}
    dd2 = d2.get("pdpData", {}) if d2 else {}
    print({**dd1, **dd2})

It outputs :

{'a': 1, 'b': 2, 'c': 3}
{'a': 2, 'b': 3}

Now that you have merged inner-dicts, all you need to do is pack them into another dict with "pdpData" as key, and pack those dicts into a list.

score 0 · Answer 2 · answered Oct 13 '17 at 09:01

0

from collections import defaultdict

d = defaultdict(dict)
for l in (l1, l2):
    for elem in l:
        d[elem['pdpData']['a']].update(elem['pdpData'])
l3 = d.values()

print(l3)

Output

dict_values([{'a': 1, 'b': 2, 'c': 3}, {'a': 2, 'b': 3}])

answered Oct 13 '17 at 09:01

Van Peer

2,127
2
25
35

You lose the order, though. – Eric Duminil Oct 13 '17 at 09:40
1

OP wants a list, which is ordered. You use a defaultdict, which is unordered. You cannot guarantee that `l3` will have the correct order. Also, you use the value `'a'` as index, which doesn't seem to be desired behaviour. – Eric Duminil Oct 13 '17 at 10:00

merging a list of nested dictionaries

2 Answers2