Python consolidate dictionary keys and values using comprehension

Question

How do I CONSOLIDATE the following using python COMPREHENSION

FROM (list of dicts)

[
 {'server':'serv1','os':'Linux','archive':'/my/folder1'}
 ,{'server':'serv2','os':'Linux','archive':'/my/folder1'}
 ,{'server':'serv3','os':'Linux','archive':'/my/folder2'}
 ,{'server':'serv4','os':'AIX','archive':'/my/folder1'}
 ,{'server':'serv5','os':'AIX','archive':'/my/folder1'}
]

TO (list of dicts with tuple as key and list of 'server#'s as value

[
 {('Linux','/my/folder1'):['serv1','serv2']}
 ,('Linux','/my/folder2'):['serv3']}
 .{('AIX','/my/folder1'):['serv4','serv5']}
]

Comprehension isn't really the tool for this job... – Shadow Jul 12 '17 at 05:54 — Shadow, Jul 12 '17 at 05:54

hiro protagonist · Accepted Answer · 2017-07-12T06:06:39.230

4

the need to be able to set default values to your dictionary and to have the same key several times may make a dict-comprehension a bit clumsy. i'd prefer something like this:

a defaultdict may help:

from collections import defaultdict

lst = [
 {'server':'serv1','os':'Linux','archive':'/my/folder1'},
 {'server':'serv2','os':'Linux','archive':'/my/folder1'},
 {'server':'serv3','os':'Linux','archive':'/my/folder2'},
 {'server':'serv4','os':'AIX','archive':'/my/folder1'},
 {'server':'serv5','os':'AIX','archive':'/my/folder1'}
]

dct = defaultdict(list)

for d in lst:
    key = d['os'], d['archive']
    dct[key].append(d['server'])

if you prefer to have a standard dictionary in the end (actually i do not really see a good reason for that) you could use dict.setdefault in order to create an empty list where the key does not yet exist:

dct = {}

for d in lst:
    key = d['os'], d['archive']
    dct.setdefault(key, []).append(d['server'])

the documentation on defaultdict (vs. setdefault):

This technique is simpler and faster than an equivalent technique using dict.setdefault()

edited Jul 12 '17 at 06:06

answered Jul 12 '17 at 05:50

hiro protagonist

44,693
14
86
111

that's not using list comprehensions, but I don't think there's a way to do that with list comprehensions because of the cumulative effect. – Jean-François Fabre Jul 12 '17 at 05:55
Do you have any proof that it is faster? – Shadow Jul 12 '17 at 05:56
@Jean-FrançoisFabre tried to find a version with comprehension first but the need to set a default value is tricky... will add a remark on that. – hiro protagonist Jul 12 '17 at 05:56
@shadow that is what the official python documentation says. did you benchmark? – hiro protagonist Jul 12 '17 at 05:57
Just a link to where the documentation states that will suffice. – Shadow Jul 12 '17 at 05:58
I'm still newbie on python collections, so for now I'll use the standard dictionary @hiroprotagonist 2nd suggestion. That's great. Thanks. – MaxGrand Jul 12 '17 at 06:36

Jean-François Fabre · Answer 2 · 2017-07-12T06:06:23.017

It's difficult to achieve with list comprehension because of the accumulation effect. However, it's possible using itertools.groupby on the list sorted by your keys (use the same key function for both sorting and grouping).

Then extract the server info in a list comprehension and prefix by the group key. Pass the resulting (group key, server list) to dictionary comprehension and here you go.

import itertools

lst = [
 {'server':'serv1','os':'Linux','archive':'/my/folder1'}
 ,{'server':'serv2','os':'Linux','archive':'/my/folder1'}
 ,{'server':'serv3','os':'Linux','archive':'/my/folder2'}
 ,{'server':'serv4','os':'AIX','archive':'/my/folder1'}
 ,{'server':'serv5','os':'AIX','archive':'/my/folder1'}
]


sortfunc = lambda x : (x['os'],x['archive'])

result = {k:[x['server'] for x in v] for k,v in itertools.groupby(sorted(lst,key=sortfunc),key = sortfunc)}


print(result)

I get:

{('Linux', '/my/folder1'): ['serv1', 'serv2'], ('AIX', '/my/folder1'): ['serv4', 'serv5'], ('Linux', '/my/folder2'): ['serv3']}

Keep in mind that it's not because it can be written in one line that it's more efficient. The defaultdict approach doesn't require sorting for instance.

Python consolidate dictionary keys and values using comprehension

2 Answers2