12

In the following code, the mc assigment works fine in Python 2 and 3.

The cc assignment, which uses the same list comprehension within a class, works in Python 2 but fails with Python 3.

What explains this behavior?

ml1 = "a b c".split()
ml2 = "1 2 3".split()
mc = [ i1 + i2 for i1 in ml1 for i2 in ml2 ]

class Foo(object):
    cl1 = ml1
    cl2 = ml2

    cc1 = [ i1 for i1 in cl1 ]
    cc2 = [ i2 for i2 in cl2 ]
    cc = [ i1 + i2 for i1 in cl1 for i2 in cl2 ]


print("mc = ", mc)
foo = Foo()
print("cc = ", foo.cc)

I get this:

(default-3.5) snafu$ python2 /tmp/z.py 
('mc = ', ['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3'])
('cc = ', ['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3'])

(default-3.5) snafu$ python3 /tmp/z.py 
Traceback (most recent call last):
  File "/tmp/z.py", line 5, in <module>
    class Foo(object):
  File "/tmp/z.py", line 11, in Foo
    cc = [ i1 + i2 for i1 in cl1 for i2 in cl2 ]
  File "/tmp/z.py", line 11, in <listcomp>
    cc = [ i1 + i2 for i1 in cl1 for i2 in cl2 ]
NameError: name 'cl2' is not defined

Why is the class variable cl2 not defined? Note that the cc2 assignment works fine, as does cc1. Swapping cl1 and cl2 in the comprehension shows that the second loop is the one that triggers the exception, not cl2 per se.)

Versions:

(default-3.5) snafu$ python2 --version
Python 2.7.11+
(default-3.5) snafu$ python3 --version
Python 3.5.1+
Reece
  • 7,616
  • 4
  • 30
  • 46

1 Answers1

5

In Python 3, list comprehensions have their own scope, which follows the same rules as a function scope. You know how the methods of a class don't automatically look inside the class scope for variable lookup?

class Example:
    var = 1
    def this_fails(self):
        print(var)
Example().this_fails()  # NameError

The same applies to any function scope nested inside a class scope, including the scope of the list comprehension. The lookup of cl2 inside the list comprehension bypasses the class scope and goes straight to the globals. It effectively works like this:

class Foo(object):
    ...
    def make_cc(outer_iterable):
        result = []
        for i1 in outer_iterable:
            for i2 in cl2:  # This fails
                result.append(i1 + i2)
        return result
    cc = make_cc(cl1)  # cl1 is evaluated outside the comprehension scope, for reasons

Note that the cl1 lookup works fine, because that happens at class scope, outside the comprehension, despite being syntactically nested inside the comprehension. They made that decision back when Python introduced genexps, because it catches a few common genexp errors earlier. It's also why the cc1 and cc2 list comprehensions work; their only use of class-level variables is in their outer (only) for iterable.

Using comprehensions and generator expressions inside a class statement is a mess. It shouldn't be, but it is. Stick to regular loops, or run the comprehensions outside the class statement so the semantics are more obvious.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • 1
    This sounds reasonable, however, it doesn't explain (to me) why `cc1 = [ i1 for i1 in cl1 ]` works but `cc = [ i1 + i2 for i1 in cl1 for i2 in cl2 ]` doesn't. Won't the scoping rules be the same? – mhawke Jul 12 '16 at 01:13
  • @mhawke: The iterable of the first `for` gets evaluated outside the comprehension scope and passed to the anonymous function as an argument, most likely for symmetry with generator expressions, which work that way because it catches a few common errors earlier. – user2357112 Jul 12 '16 at 01:16
  • "Using comprehensions and generator expressions inside a class statement is a mess." I agree: The successful `cc1` and `cc2` assignments and failed `cc` assignment seem uncharacteristically surprising and unintuitive. – Reece Jul 12 '16 at 16:42
  • Consider `itertools.product`. The above comprehension becomes `cc = [ i1 + i2 for i1, i2 in itertools.product(cl1, cl2) ]`, which *does* work within classes for the reasons given in the above answer. – Reece Jul 12 '16 at 17:55