python: iterating through a dictionary with list values

Question

Given a dictionary of lists, such as

d = {'1':[11,12], '2':[21,21]}

Which is more pythonic or otherwise preferable:

for k in d:
    for x in d[k]:
        # whatever with k, x

or

for k, dk in d.iteritems():
    for x in dk:
        # whatever with k, x

or is there something else to consider?

EDIT, in case a list might be useful (e.g., standard dicts don't preserve order), this might be appropriate, although it's much slower.

d2 = d.items()
for k in d2:
        for x in d2[1]:
            # whatever with k, x

sorry just got busy ... @foosion is was talking about something similar to kelorecs answer below with list comprehensions .. not so readable though — woofmeow, Aug 17 '13 at 14:36

Brionius · Answer 1 · 2013-08-17T14:57:41.890

20

Here's a speed test, why not:

import random
numEntries = 1000000
d = dict(zip(range(numEntries), [random.sample(range(0, 100), 2) for x in range(numEntries)]))

def m1(d):
    for k in d:
        for x in d[k]:
            pass

def m2(d):
    for k, dk in d.iteritems():
        for x in dk:
            pass

import cProfile

cProfile.run('m1(d)')

print

cProfile.run('m2(d)')

# Ran 3 trials:
# m1: 0.205, 0.194, 0.193: average 0.197 s
# m2: 0.176, 0.166, 0.173: average 0.172 s

# Method 1 takes 15% more time than method 2

cProfile example output:

         3 function calls in 0.194 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.194    0.194 <string>:1(<module>)
        1    0.194    0.194    0.194    0.194 stackoverflow.py:7(m1)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}



         4 function calls in 0.179 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.179    0.179 <string>:1(<module>)
        1    0.179    0.179    0.179    0.179 stackoverflow.py:12(m2)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}

edited Aug 17 '13 at 14:57

answered Aug 17 '13 at 14:21

Brionius

13,858
3
38
49

3

iteritems() is a bit faster and has 'iter' in its name. What more could one ask? :-) – foosion Aug 17 '13 at 14:26
On my machine, I'm getting 0.172 for m1 and 0.185 for m2 using your code. – foosion Aug 17 '13 at 14:48
How strange - I tried it a few more times, and m1 consistently takes around 15% more time on my machine. Python 2.7, Intel i5. – Brionius Aug 17 '13 at 14:54
Python 2.7.2, i5, win 7. Very odd. I ran your exact code, other than indenting pass. – foosion Aug 17 '13 at 14:57
Yeah, I just fixed the indenting, but same results. – Brionius Aug 17 '13 at 14:58
Added some cProfile output, in case that helps solve the mystery. – Brionius Aug 17 '13 at 15:02
Continues to be a mystery - see the results I posted in an answer – foosion Aug 17 '13 at 15:06
Mac OS X 10.7.5?? You? – Brionius Aug 17 '13 at 15:07

score 10 · Answer 2 · answered Nov 07 '14 at 00:02

I considered a couple methods:

import itertools

COLORED_THINGS = {'blue': ['sky', 'jeans', 'powerline insert mode'],
                  'yellow': ['sun', 'banana', 'phone book/monitor stand'],
                  'red': ['blood', 'tomato', 'test failure']}

def forloops():
    """ Nested for loops. """
    for color, things in COLORED_THINGS.items():
        for thing in things:
            pass

def iterator():
    """ Use itertools and list comprehension to construct iterator. """
    for color, thing in (
        itertools.chain.from_iterable(
            [itertools.product((k,), v) for k, v in COLORED_THINGS.items()])):
        pass

def iterator_gen():
    """ Use itertools and generator to construct iterator. """
    for color, thing in (
        itertools.chain.from_iterable(
            (itertools.product((k,), v) for k, v in COLORED_THINGS.items()))):
        pass

I used ipython and memory_profiler to test performance:

>>> %timeit forloops()
1000000 loops, best of 3: 1.31 µs per loop

>>> %timeit iterator()
100000 loops, best of 3: 3.58 µs per loop

>>> %timeit iterator_gen()
100000 loops, best of 3: 3.91 µs per loop

>>> %memit -r 1000 forloops()
peak memory: 35.79 MiB, increment: 0.02 MiB

>>> %memit -r 1000 iterator()
peak memory: 35.79 MiB, increment: 0.00 MiB

>>> %memit -r 1000 iterator_gen()
peak memory: 35.79 MiB, increment: 0.00 MiB

As you can see, the method had no observable impact on peak memory usage, but nested for loops were unbeatable for speed (not to mention readability).

score 3 · Answer 3 · answered Aug 17 '13 at 14:26

3

Here's the list comprehension approach. Nested...

r = [[i for i in d[x]] for x in d.keys()]
print r

[[11, 12], [21, 21]]

answered Aug 17 '13 at 14:26

kelorek

6,042
6
29
32

For something like that, d.items() seems better, at least to me, especially if you want to do something with both the keys and the values. – foosion Aug 17 '13 at 14:31
Well, right.. I don't really know what your use case is. You asked @woofmeow for clarification on list comprehensions. – kelorek Aug 17 '13 at 14:44
No worries. I suppose what I was really asking was how a list comprehension was responsive to a question which (as edited) wants to do something with the key and the values. – foosion Aug 17 '13 at 14:51

score 2 · Answer 4 · answered Aug 17 '13 at 15:04

My results from Brionius code:

         3 function calls in 0.173 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.173    0.173 <string>:1(<module>)
        1    0.173    0.173    0.173    0.173 speed.py:5(m1)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Prof
iler' objects}


         4 function calls in 0.185 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.185    0.185 <string>:1(<module>)
        1    0.185    0.185    0.185    0.185 speed.py:10(m2)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Prof
iler' objects}
        1    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' obje
cts}

python: iterating through a dictionary with list values

4 Answers4

Linked