list comprehension in exec with empty locals: NameError

Question

Consider the following snippet:

def bar():
    return 1
print([bar() for _ in range(5)])

It gives an expected output [1, 1, 1, 1, 1].

However, if I try to exec the same snippet in empty environment (locals and globals both are set to {}), it gives NameError:

if 'bar' in globals() or 'bar' in locals():
    del bar
# make sure we reset settings

exec("""
def bar():
    return 1
print([bar() for _ in range(5)])
""", {}, {})

NameError: name 'bar' is not defined

If I invoke exec like exec(…, {}) or exec(…), it is executed as expected.

Why?

EDIT:

Consider also the following snippet:

def foo():
    def bar():
        return 1
    print('bar' in globals()) # False
    print('bar' in locals()) # True
    print(['bar' in locals() for _ in [1]]) # [False]
    print([bar() for _ in [1, 2]]) # [1, 1]

Just like in my first exec, we don't have bar in locals inside list comprehension. However, if we try to invoke it, it works!

Answer 1

The solution to your problem lies here:

In all cases, if the optional parts are omitted, the code is executed in the current scope. If only globals is provided, it must be a dictionary, which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

https://docs.python.org/3/library/functions.html#exec

Basically, your problem is that bar is defined in the scope of locals and only in locals. Therefore, this exec() statement works:

exec("""
def bar():
    return 1
print(bar())
""", {}, {})

The list comprehension however creates a new local scope, one in which bar is not defined and can therefore not be looked up.

This behaviour can be illustrated with:

exec("""
def bar():
    return 1
print(bar())
print(locals())
print([locals() for _ in range(1)])
""", {}, {})

which returns

1
{'bar': <function bar at 0x108efde18>}
[{'_': 0, '.0': <range_iterator object at 0x108fa8780>}]

EDIT

In your original example, the definition of bar is found in the (module level) global scope. This corresponds to

Remember that at module level, globals and locals are the same dictionary.

In the exec example, you introduce an artificial split in scopes between globals and locals by passing two different dictionaries. If you passed the same one or only the globals one (which would in turn mean that this one will be used for both globals and locals) , your example would also work.

As for the example introduced in the edit, this boils down to the scoping rules in python. For a detailed explanation, please read: https://docs.python.org/3/tutorial/classes.html#python-scopes-and-namespaces

In short, while bar is not in the local scope of the list comprehension and neither in the global scope, it is in the scope of foo. And given Python scoping rules, if a variable is not found in the local scope, it will be searched for in the enclosing scopes until the global scope is reached. In your example, foo's scope sits between the local scope and the global scope, so bar will be found before reaching the end of the search.

This is however still different to the exec example, where the locals scope you pass in is not enclosing the scope of the list comprehension, but completely divided from it.

Another great explanation of scoping rules including illustrations can be found here: http://sebastianraschka.com/Articles/2014_python_scope_and_namespaces.html

Answer 2

As Hendrik Makait found out, the exec documentation says that

If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

You can get the same behaviour by embedding the code into a class definition:

class Foo:
    def bar():
        return 1
    print([bar() for _ in range(5)])

Run it in Python 3 and you will get

Traceback (most recent call last):
  File "foo.py", line 9, in <module>
    class Foo:
  File "foo.py", line 15, in Foo
    print({bar() for _ in range(5)})
  File "foo.py", line 15, in <setcomp>
    print({bar() for _ in range(5)})
NameError: global name 'bar' is not defined

The reason for the error is as Hendrik said that a new implicit local scope is created for list comprehensions. However Python only ever looks names up in 2 scopes: global or local. Since neither the global nor the new local scope contains the name bar, you get the NameError.

The code works in Python 2, because list comprehensions have a bug in Python 2 in that they do not create a new scope, and thus they leak variables into their current local scope:

class Foo:
    [1 for a in range(5)]
    print(locals()['a'])

Run it in Python 2 and the output is 4. The variable a is now within the locals in the class body, and retains the value from the last iteration. In Python 3 you will get a KeyError.

You can get the same error in Python 2 too though, if you use a generator expression, or a dictionary/set comprehension:

class Foo:
    def bar():
        return 1
    print({bar() for _ in range(5)})

The error can be produced also by just using simply

class Foo: 
    bar = 42
    class Bar:
        print(bar)

This is unlike

def foo():
    bar = 42
    def baz():
        print(bar)
    baz()

because upon execution of foo, Python makes baz into a closure, which will access the bar variable via a special bytecode instruction.

Answer 3

I'm late to the party here, but there is a better documentation reference buried in the execution model.

In section 4.2.2 Resolution of names:

Class definition blocks and arguments to exec() and eval() are special in the context of name resolution. ...

And then in 4.2.4 Interaction with dynamic features:

The eval() and exec() functions do not have access to the full environment for resolving names. Names may be resolved in the local and global namespaces of the caller. Free variables are not resolved in the nearest enclosing namespace, but in the global namespace. [1] The exec() and eval() functions have optional arguments to override the global and local namespace. If only one namespace is specified, it is used for both.

[1] This limitation occurs because the code that is executed by these operations is not available at the time the module is compiled.

Answer 4

Here's a solution!

We needed to get the local namespace out after the exec() to track modifications. This doesn't work with only one namespace, so we did this:

 class MagickNameSpace(UserDict, dict):
    """A magic namespace for Python 3 exec().
    We need separate global and local namespaces in exec(). This does not
    work well in Python 3, because in Python 3 the enclosing namespaces are
    not used to look up variables, which seems to be an optimization thing
    as the exec'd code isn't available at module compilation.

    So we make a MagickNameSpace that stores all new variables in a
    separate dict, conforming to the local/enclosing namespace, but
    looks up variables in both.
    """


    def __init__(self, ns, *args, **kw):
        UserDict.__init__(self, *args, **kw)
        self.globals = ns

    def __getitem__(self, key):
        try:
            return self.data[key]
        except KeyError:
            return self.globals[key]

    def __contains__(self, key):
        return key in self.data or key in self.globals

Replace the old code:

exec(code, global_ns, local_ns)
return local_ns

with:

ns = MagickNameSpace(global_ns)
ns.update(local_ns)
exec(code, ns)
return ns.data

Answer 5

Edit

To answer your edited question, user @Hendrik Makait said bar is not in the scope of list comprehension:

def foo():
    def bar():
        return 1
    print('bar' in globals()) # False, because the scope of foo and bar are diferents, foo is globals() scope, bar are in the scope of foo
    print('bar' in locals()) # True
    print(['bar' in locals() for _ in [1]]) # [False], because a new implicit scope is defined in list comprehension, as user @Antti Haapala said
    print([bar() for _ in [1, 2]]) # [1, 1]

To answer the original question:

If you create two different dictionaries, it wont recognize the local and globals definitions, the variables are not updated as @PM 2Ring said:

exec("""
def bar():
    return 1
print(bar())
print("bar" in globals())
print("bar" in locals())
print([bar() for _ in range(5)])
""", {},{})

it prints:

1
False #not in globals
True
Traceback (most recent call last):
  File "python", line 17, in <module>
  File "<string>", line 7, in <module>
  File "<string>", line 7, in <listcomp>
NameError: name 'bar' is not defined

A way to do it, is update the variables, like this globals().update(locals()):

exec("""
def bar():
    return 1
globals().update(locals())
print("bar" in globals())
print("bar" in locals())
print([bar() for _ in range(5)])
""", {}, {})

wich gives:

True
True
[1, 1, 1, 1, 1]

But, if you remove the dictionaries, or create one and give it to the exec function as same parameter, it works:

d={}

exec("""
def bar():
    return 1
print("bar" in globals())
print("bar" in locals())
print([bar() for _ in range(5)])
""",d,d)

it prints:

True
True
[1, 1, 1, 1, 1]

That's why you get the error, it could't find your function in the globals

Or simply, don't give the parameters:

exec("""
def bar():
    return 1
print(bar())
print("bar" in globals())
print("bar" in locals())
print([bar() for _ in range(5)])
""")

Cause the same efect.