Python dictionary union not working as expected with dictionaries of depth greater than 1

Question

Here is an example to illustrate the problem...

a = {
    "foo" : 2,
    "bar" : 3,
}

b = {
    "bar" : 4,
    "zzz" : 5,
}

print(json.dumps(dict(a, **b), indent=4))

This gives you the following result...

{
    "foo": 2,
    "bar": 4,
    "zzz": 5
}

Notice how the "foo" key from a was added to the result?

Now look at this example...

a = {
    "foo" : {
        "1" : {
            "foo" : True,
        },
        "2" : {
            "foo" : True,
        },
        "3" : {
            "foo" : True,
        },
        "4" : {
            "foo" : True
        }
    }
}

b = {
    "foo" : {
        "1" : {
            "foo" : True,
        },
        "2" : {
            "foo" : True,
        },
        "3" : {
            "foo" : False,
        }
    }
}

print(json.dumps(dict(a, **b), indent=4))

This gives you the following result...

{
    "foo": {
        "1": {
            "foo": true
        },
        "3": {
            "foo": false
        },
        "2": {
            "foo": true
        }
    }
}

"3" was updated just like how "bar" was updated in the previous example but notice how the "4" key from a was not added to the result like how "foo" was in the previous example?

So my expected output would be:

{
    "foo": {
        "1": {
            "foo": true
        },
        "3": {
            "foo": false
        },
        "2": {
            "foo": true
        },
        "4": {
            "foo": true
        }
    }
}

How can I modify this dictionary union process so that keys in a that are not in b are retained?

My overall goal is to have my dictionary a but with any values that are in b overridden.

So what are you expecting to get? Give us an expected output for your current example. — Taku, Sep 22 '17 at 02:55
@abccd I'm expecting to get that final result I posted but with the `"4"` key in it as well. — Ogen, Sep 22 '17 at 03:17
IIRC the folks at Webpack have a config merger which acts on js objects which look quite a lot like dicts. Don't let the config/webpack trappings fool you, it's really a deep merger. I'd love to see it in python. https://github.com/survivejs/webpack-merge — JL Peyret, Sep 22 '17 at 13:58

score 2 · Answer 1 · answered Sep 22 '17 at 03:00

Let's talk about this behave. I think the critical point is how **kwargs works.

In your first case, when you pass **b to dict(), it is the same as dict(a, bar=4, zzz=5). So under this situation, {zzz: 5} was added into dictionary, and {bar: 4} was updated.

But in your second case, when you pass **b to dict(), it is the same as dict(a, foo={...}). As the foo parameter override the same key in a, the result is absolutely the same as b. So it is not an update action but an override action.

Take another example:

a = {
    "foo" : {
        "1" : {
            "foo" : True,
        },
        "2" : {
            "foo" : True,
        },
        "3" : {
            "foo" : True,
        },
        "4" : {
            "foo" : True
        }
    }
}

b = {
    "foo" : {
        "1" : {
            "foo" : True,
        },
        "2" : {
            "foo" : True,
        },
        "3" : {
            "foo" : False,
        },
        "5" : {
            "foo" : False,
        }
    }
}

print(json.dumps(dict(a, **b), indent=4))

output:
{
    "foo": {
        "1": {
            "foo": true
        },
        "2": {
            "foo": true
        },
        "3": {
            "foo": false
        },
        "5": {
            "foo": false
        }
    }
}

Oh I see, so the `**` operator is not recursive, it only operates on the top level keys. I can see why its not working but this isn't exactly a solution to my overall problem. — Ogen, Sep 22 '17 at 03:18
@Ogen I know what you really want, it is `deep merge` which is a common topic, just search it or see duplicated question in your question's comment. — Sraw, Sep 22 '17 at 03:20

Taku · Answer 2 · 2017-09-22T03:42:06.497

For your specific case, that there's only a dict depth of one, you can just use a loop and call dict.update() to update the values. The reason your's doesn't work is because dict(a, **b) overrides all the duplicate keys a and b shared in common and only keeps b's, which means that a's "foo" was completely replaced by b's "foo"

import json

for k in a.keys():
    try:
        a[k].update(b[k])
    except KeyError:
        continue

print(json.dumps(a, indent=4))

This will achieve your desired result "1" ,"2" ,"3" being updated from dict b and "4" being kept from dict a. But if you have a dict with n depth, that will not exactly work. You will need to use a recursive function, as followed:

Sample data:

a = {
    "foo" : {
        "1" : {
            "foo" : {"1": True,
                     "2": False},
        },
        "2" : {
            "foo" : True,
        },
        "3" : {
            "foo" : True,
        },
        "4" : {
            "foo" : True
        }
    },
}

b = {
    "foo" : {
        "1" : {
            "foo" : {"1": True,
                     "3": True},
        },
        "2" : {
            "foo" : True,
        },
        "3" : {
            "foo" : False,
        }
    }
}

Recursive code:

import json

def update(dctA, dctB):
    if isinstance(dctA, dict) and isinstance(dctB, dict):
        for k in set(dctA.keys()) & set(dctB.keys()):
            update(dctA[k], dctB[k])
            try:
                dctA[k].update(dctB[k])
                dctB.pop(k)
            except (KeyError, AttributeError):
                continue
    return dctA

print(json.dumps(update(a, b), indent=4))

The output will be:

{
    "foo": {
        "1": {
            "foo": {
                "1": true,
                "2": false,
                "3": true
            }
        },
        "2": {
            "foo": true
        },
        "3": {
            "foo": false
        },
        "4": {
            "foo": true
        }
    }
}

Which as you can see, the "3" key got updated with "foo" changing to false, and "4" being kept as dict b doesn't update key "4". Furthermore, thanks to the recursion function, the nested "foo" key also been updated, {"1": True, "2": False} with {"1": True, "3": True} updated to {"1": true, "2": false, "3": true}. The "1" stayed the same, "2" was kept because dict b does not have key "2", and key "3" was created because it is present in dict b.

Python dictionary union not working as expected with dictionaries of depth greater than 1

2 Answers2