Python 3.8: Having difficulty with searching through nested lists and dictionaries

Question

I am accessing data via an API and I end up with data in nested dictionaries and lists.

I am trying to get a list of the locations by searching for the "unitaryAuthArea" in this nested set of data.

The only result I get is: Sites is a dictionary.

Python 3.8

sites={'Locations': {'Location':[{'elevation': '50.0', 'id': '14', 'latitude': '54.9375', 'longitude': '-2.8092', 'name': 'Carlisle Airport', 'region': 'nw', 'unitaryAuthArea': 'Cumbria'},{'elevation': '108.0', 'id': '355874', 'latitude': '52.415775', 'longitude': '-4.059387', 'name': 'Penglais School', 'region': 'wl', 'unitaryAuthArea': 'Ceredigion'},{'elevation': '75.0', 'id': '3930', 'latitude': '51.55138', 'longitude': '-2.55933', 'name': 'Almondsbury', 'region': 'sw', 'unitaryAuthArea': 'South Gloucestershire'},{'elevation': '22.0', 'id': '26', 'latitude': '53.3336', 'longitude': '-2.85', 'name': 'Liverpool John Lennon Airport', 'region': 'nw', 'unitaryAuthArea': 'Merseyside'}]}}
 
srch='unitaryAuthArea'
 
def nested_extract(nested_sites, key):
    for k, v in nested_sites.items():
        if isinstance(k, dict):
            print('\n K is a dictionary')
            nested_extract(k,key)
        if isinstance(k, list):
            print('\n K is a list')
            if isinstance([0], dict):
                print('\n K is an Inner dictionary')
                nested_extract([0],key)
        if k == key:
            print(v)
         
 
if isinstance(sites, dict):
    print('\n Sites is a dictionary')
    nested_extract(sites,srch)
else:
    print('\n Sites is NOT a dictionary')

Many thanks. Dave

At least, you should check if it's a dictionary or a list, not key but value, I think. — Daria Pydorenko, Jul 09 '20 at 09:03
Your recursion does not run even a second time because `k` is a `string` (with value `"Locations"`). And since it is not equal to `unitaryAuthArea`, it does not print anything in the function. After all, check this question: https://stackoverflow.com/q/10756427/4636715 — vahdet, Jul 09 '20 at 09:09

score 1 · Accepted Answer · answered Jul 09 '20 at 09:06

Changed the loop a bit, to check v instead of k. And loop if list.

sites={'Locations': {'Location':[{'elevation': '50.0', 'id': '14', 'latitude': '54.9375', 'longitude': '-2.8092', 'name': 'Carlisle Airport', 'region': 'nw', 'unitaryAuthArea': 'Cumbria'},{'elevation': '108.0', 'id': '355874', 'latitude': '52.415775', 'longitude': '-4.059387', 'name': 'Penglais School', 'region': 'wl', 'unitaryAuthArea': 'Ceredigion'},{'elevation': '75.0', 'id': '3930', 'latitude': '51.55138', 'longitude': '-2.55933', 'name': 'Almondsbury', 'region': 'sw', 'unitaryAuthArea': 'South Gloucestershire'},{'elevation': '22.0', 'id': '26', 'latitude': '53.3336', 'longitude': '-2.85', 'name': 'Liverpool John Lennon Airport', 'region': 'nw', 'unitaryAuthArea': 'Merseyside'}]}}

srch='unitaryAuthArea'

def nested_extract(nested_sites, key):
    for k, v in nested_sites.items():
        if isinstance(v, dict):
            print('\n V is a dictionary')
            nested_extract(v, key)
        if isinstance(v, list):
            print('\n V is a list')
            for elem in v:
                nested_extract(elem, key)
        if k == key:
            print(v)

if isinstance(sites, dict):
    print('\n Sites is a dictionary')
    nested_extract(sites,srch)
else:
    print('\n Sites is NOT a dictionary')

Thanks, this works well as it is not constrained to this specific structure. — Dave, Jul 09 '20 at 10:49

score 1 · Answer 2 · answered Jul 09 '20 at 09:12

Is this what you are looking for as you are saying that you want to get a list of the locations by searching for the "unitaryAuthArea"? This only works if the structure remains the same. I added a if statement in the list comprehension to make sure you will not get an error in case your keyword "unitaryAuthArea" is not in the dict. Hope it helps you. If you are searching for something else, just give me comment.

sites ={'Locations': {'Location' :
    [{'elevation': '50.0', 'id': '14', 'latitude': '54.9375', 'longitude': '-2.8092', 'name': 'Carlisle Airport', 'region': 'nw', 'unitaryAuthArea': 'Cumbria'}
    ,{'elevation': '108.0', 'id': '355874', 'latitude': '52.415775', 'longitude': '-4.059387', 'name': 'Penglais School', 'region': 'wl', 'unitaryAuthArea': 'Ceredigion'}
    ,{'elevation': '75.0', 'id': '3930', 'latitude': '51.55138', 'longitude': '-2.55933', 'name': 'Almondsbury', 'region': 'sw', 'unitaryAuthArea': 'South Gloucestershire'}
    ,{'elevation': '22.0', 'id': '26', 'latitude': '53.3336', 'longitude': '-2.85', 'name': 'Liverpool John Lennon Airport', 'region': 'nw', 'unitaryAuthArea': 'Merseyside'}]}}

srch ='unitaryAuthArea'

results = [location['unitaryAuthArea'] for location in sites['Locations']['Location'] if location.get('unitaryAuthArea', False)]

print(results)
#output: ['Cumbria', 'Ceredigion', 'South Gloucestershire', 'Merseyside']

score 0 · Answer 3 · answered Jul 09 '20 at 09:05

You should iterate by value not by key:

if isinstance(v, dict):
      print('\n V is a dictionary')
      nested_extract(v, key)
if isinstance(v, list):
      print('\n V is a list')
      if isinstance(v[0], dict):
           print('\n V is an Inner dictionary')
           nested_extract(v[0], key)

And then instead of [0] iterate by each value of the list:

if isinstance(v, dict):
    print('\n V is a dictionary')
    nested_extract(v, key)
if isinstance(v, list):
    print('\n V is a list')
    for x in v:
        if isinstance(x, dict):
            print('\n X is an Inner dictionary')
            nested_extract(x, key)

Python 3.8: Having difficulty with searching through nested lists and dictionaries

3 Answers3