When is the existence of nonlocal variables checked?

Question

I am learning Python and right now I am on the topic of scopes and nonlocal statement. At some point I thought I figured it all out, but then nonlocal came and broke everything down.

Example number 1:

print( "let's begin" )
def a():
    def b():
        nonlocal x
        x = 20
    b()

a()

Running it naturally fails.
What is more interesting is that print() does not get executed. Why?.

My understanding was that enclosing def a() is not executed until print() is executed, and nested def b() is executed only when a() is called. I am confused...

Ok, let's try example number 2:

print( "let's begin" )
def a():
    if False: x = 10
    def b():
        nonlocal x
        x = 20
    b()

a()

Aaand... it runs fine. Whaaat?! How did THAT fix it? x = 10 in function a is never executed!

My understanding was that nonlocal statement is evaluated and executed at run-time, searching enclosing function's call contexts and binding local name x to some particular "outer" x. And if there is no x in outer functions - raise an exception. Again, at run-time.

But now it looks like this is done at the time of syntax analysis, with pretty dumb check "look in outer functions for x = blah, if there is something like this - we're fine," even if that x = blah is never executed...

Can anybody explain me when and how nonlocal statement is processed?

Answer 1

You can see what the scope of b knows about free variables (available for binding) from the scope of a, like so:

import inspect

print( "let's begin" )

def a():
    if False:
        x = 10

    def b():
        print(inspect.currentframe().f_code.co_freevars)
        nonlocal x
        x = 20

    b()

a()

Which gives:

let's begin
('x',)

If you comment out the nonlocal line, and remove the if statement with x inside, the you'll see the free variables available to b is just ().

So let's look at what bytecode instruction this generates, by putting the definition of a into IPython and then using dis.dis:

In [3]: import dis

In [4]: dis.dis(a)
  5           0 LOAD_CLOSURE             0 (x)
              2 BUILD_TUPLE              1
              4 LOAD_CONST               1 (<code object b at 0x7efceaa256f0, file "<ipython-input-1-20ba94fb8214>", line 5>)
              6 LOAD_CONST               2 ('a.<locals>.b')
              8 MAKE_FUNCTION            8
             10 STORE_FAST               0 (b)

 10          12 LOAD_FAST                0 (b)
             14 CALL_FUNCTION            0
             16 POP_TOP
             18 LOAD_CONST               0 (None)
             20 RETURN_VALUE

So then let's look at how LOAD_CLOSURE is processed in ceval.c.

TARGET(LOAD_CLOSURE) {
    PyObject *cell = freevars[oparg];
    Py_INCREF(cell);
    PUSH(cell);
    DISPATCH();
}

So we see it must look up x from freevars of the enclosing scope(s).

This is mentioned in the Execution Model documentation, where it says:

The nonlocal statement causes corresponding names to refer to previously bound variables in the nearest enclosing function scope. SyntaxError is raised at compile time if the given name does not exist in any enclosing function scope.

Answer 2

First, understand that python will check your module's syntax and if it detects something invalid it raises a SyntaxError which stops it from running at all. Your first example raises a SyntaxError but to understand exactly why is pretty complicated although it is easier to understand if you know how __slots__ works so I will quickly introduce that first.

---

When a class defines __slots__ it is basically saying that the instances should only have those attributes so each object is allocated memory with space for only those, trying to assign other attributes raises an error

class SlotsTest:
    __slots__ = ["a", "b"]

x = SlotsTest()

x.a = 1 ; x.b = 2
x.c = 3 #AttributeError: 'SlotsTest' object has no attribute 'c'

The reason x.c = 3 can't work is that there is no memory space to put a .c attribute in.

If you do not specify __slots__ then all instances are created with a dictionary to store the instance variables, dictionaries do not have any limitations on how many values they contain

class DictTest:
    pass

y = DictTest()
y.a = 1 ; y.b = 2 ; y.c = 3
print(y.__dict__) #prints {'a': 1, 'b': 2, 'c': 3}

---

Python functions work similar to slots. When python checks the syntax of your module it finds all variables assigned (or attempted to be assigned) in each function definition and uses that when constructing frames during execution.

When you use nonlocal x it gives an inner function access to a specific variable in the outer function scope but if there is no variable defined in the outer function then nonlocal x has no space to point to.

Global access doesn't run into the same issue since python modules are created with a dictionary to store its attributes. So global x is allowed even if there is no global reference to x