Closures in list comprehension in Python

Question

While playing with closures in list comprehension, I found this:

xs = [1, 2, 3]
fs = [lambda: _ for _ in xs]
vs = [_() for _ in fs]
print vs           # [<function <lambda> at 0x020324B0>, <function <lambda> at 0x020D6AB0>, <function <lambda> at 0x020D6AF0>]
print vs[0]        # <function <lambda> at 0x020324B0>
print vs[0]()      # <function <lambda> at 0x020D6AF0>
print vs[0]()()    # <function <lambda> at 0x020D6AF0>
print vs[0]()()()  # <function <lambda> at 0x020D6AF0>

Shouldn't vs containing ints instead of lambdas?

But if we use different names in the two list comprehensions it will work as expected:

xs = [1, 2, 3]
fs = [lambda: x for x in xs]
vs = [f() for f in fs]

I tested it in Python2.7.5 and Python2.7.8, both gives the same results. (Python3.4.1 work as expected) Is there an explanation for it or is it a bug?

The namespace handling changed between Python 2 and 3; list comprehensions are not in a separate scope in Python 2. — Martijn Pieters, Aug 10 '14 at 09:26
This is behavior that the core devs eventually decided was confusing and undesirable, so they changed it in 3.x. But it's not really a "bug", and there's some rare code that usefully relies on it, so they didn't change it in 2.7. (Also, they weren't planning on backporting the listcomp performance optimizations to 2.x, and changing scoping without that backport would make listcomps about 50% slower, which no one would be happy with.) — abarnert, Aug 10 '14 at 10:40
As a side note, because `_` is rebound to the last expression evaluated in the interactive interpreter, it's usually not a good idea to use `_` for tests like this if you're going to run them interactively, because it just leads to one more thing to get confused about. It's also not good to use `_` for cases like this in the first place, because idiomatically it means "I'm not going to use this value," but you're using it. — abarnert, Aug 10 '14 at 10:41

score 6 · Accepted Answer · edited Jul 04 '18 at 23:16

There are two things going on:

Closures bind late; when you call the lambda, it'll take _ from the closed over namespace. It'll not take the value of _ at the time you created the lambda object.

See What do (lambda) function closures capture?
Python 2 doesn't use a separate namespace for list comprehensions, Python 3 does.

See Python list comprehension rebind names even after scope of comprehension. Is this right?

As such, _ is taken from the namespace you execute the list comprehension in, and by the time you execute a lambda in the vs list comprehension, _ is bound to the very lambda you are trying to execute (as the loop target), not the last value from xs.

In Python 3 it works because a new namespace is used for all names except the original input iterable to the list comprehension. There _ lives in the new scope of the list comprehension and thus remains bound to 3.

By using a different name for the loop target in your second attempt, you are not masking the _ closed over name, leaving _ still bound to 3.

Closures in list comprehension in Python

1 Answers1