Remove all None values from nested dictionaries

Question

I have the following nested dictionary in json example:

{
  "DICT": {
    "List of dict": [
      { #first dict inside the outer list
        "K1": "V1",
        "K2": "V2",
        "K3": "V3",
        "K4": [
          {
            "K4_1_1": "V4_1"
          },
          {
            "K4_2_1": "V4_2"
          },
          {
            "K4_3_1": null
          }
        ],
        "K5 is a list of Dict": [
          {
            "K5_1_1": "V5_1",
            "K5_1_2": "V5_2",
            "K5_1_3": "V5_3",
            "K5_1_4": "V5_4"
          },
          {
            "K5_2_1": "V5_1",
            "K5_2_2": "V5_2",
            "K5_2_3": "V5_3",
            "K5_2_4": "V5_4"
          }
        ]
      },
      { #second dict in the outerlist
        "K1": "V1",
        "K2": "V2",
        "K3": "V3",
        "K4": [
          {
            "K4_1_1": "V4_1_1"
          },
          {
            "K4_2_1": "V4_2_1"
          }
        ],
        "K5": {
          "K5_1_1": "V_1_1",
          "K5_1_2": "V_1_2",
          "K5_1_3": null,
          "K5_1_4": null
        }
      }
    ]
  }
}

Note that, K4 and K5 are always list of dict. I need to get rid of all null no matter how deep they are inside the dictionary or inside a list. So I wrote the following python function, however the output is the same and all None values are still there:

def RemoveNones(Dict):
    for k, v in Dict.items():
        if type(v) == collections.OrderedDict:
            RemoveNones(v)
        elif type(v) == list:
            for i in v:
                RemoveNones(i)
        else:
            Dict = dict((K,V) for K,V in Dict.items() if V!=None)

my dictionaries inside are not dict but <class 'collections.OrderedDict'>.

Richard Inglis · Accepted Answer · 2018-06-26T11:29:34.330

The syntax for type() is like this: if type(v) is list: (not ==)

So you want something like this:

import json
from collections import OrderedDict

raw_text = '{"DICT":{"List of dict":[{"K1":"V1","K2":"V2","K3":"V3","K4":[{"K4_1_1":"V4_1"},{"K4_2_1":"V4_2"},{"K4_3_1":null}],"K5 is a list of Dict":[{"K5_1_1":"V5_1","K5_1_2":"V5_2","K5_1_3":"V5_3","K5_1_4":"V5_4"},{"K5_2_1":"V5_1","K5_2_2":"V5_2","K5_2_3":"V5_3","K5_2_4":"V5_4"}]},{"K1":"V1","K2":"V2","K3":"V3","K4":[{"K4_1_1":"V4_1_1"},{"K4_2_1":"V4_2_1"}],"K5":{"K5_1_1":"V_1_1","K5_1_2":"V_1_2","K5_1_3":null,"K5_1_4":null}}]}}'

raw_json = json.JSONDecoder(object_pairs_hook=OrderedDict).decode(raw_text)

def remove_nulls(x):
    if type(x) is list:
        return [remove_nulls(v) for v in x if v is not None]
    elif type(x) is OrderedDict:
        return OrderedDict((k,remove_nulls(v)) for k,v in x.items() if v is not None)
    else:
        return x

de_nullified_json = remove_nulls(raw_json)
print(json.dumps(de_nullified_json, indent=2))

Output:

{
  "DICT": {
    "List of dict": [
      {
        "K1": "V1",
        "K2": "V2",
        "K3": "V3",
        "K4": [
          {
            "K4_1_1": "V4_1"
          },
          {
            "K4_2_1": "V4_2"
          },
          {}
        ],
        "K5 is a list of Dict": [
          {
            "K5_1_1": "V5_1",
            "K5_1_2": "V5_2",
            "K5_1_3": "V5_3",
            "K5_1_4": "V5_4"
          },
          {
            "K5_2_1": "V5_1",
            "K5_2_2": "V5_2",
            "K5_2_3": "V5_3",
            "K5_2_4": "V5_4"
          }
        ]
      },
      {
        "K1": "V1",
        "K2": "V2",
        "K3": "V3",
        "K4": [
          {
            "K4_1_1": "V4_1_1"
          },
          {
            "K4_2_1": "V4_2_1"
          }
        ],
        "K5": {
          "K5_1_1": "V_1_1",
          "K5_1_2": "V_1_2"
        }
      }
    ]
  }
}

Thank you for the explanation and the answer, this works. I was watching your edits why did we move from: `json.load(f)` to `json.JSONDecoder(object_pairs_hook=OrderedDict).decode(raw_text)` ? Is it to make the output same order as the input ? As for my `==` and `!=` I come from a C++ background. — Hadi Farah, Jun 26 '18 at 11:37
`json.load(f)` generates regular `dict` objects (not `OrderedDicts`) so you lose the order of the elements. — Richard Inglis, Jun 26 '18 at 11:38
But is the output correct? some of the elements are empty `{ }` — Richard Inglis, Jun 26 '18 at 11:38
yes I can deal with empty `{}` because at some point later on I use `gettext()` which knows how to deal with empty brackets but not `None` values — Hadi Farah, Jun 26 '18 at 11:39
There's a good explanation of `is` vs `==` here: https://stackoverflow.com/questions/14247373/python-none-comparison-should-i-use-is-or — Richard Inglis, Jun 26 '18 at 11:49

Remove all None values from nested dictionaries

1 Answers1