In Python 2, how do I write to variable in the parent scope?

Question

I have some code like:

def example():
    # other logic omitted

    stored_blocks = {}
    def replace_blocks(m):
        block = m.group(0)
        block_hash = sha1(block)
        stored_blocks[block_hash] = block
        return '{{{%s}}}' % block_hash

    num_converted = 0
    def convert_variables(m):
        name = m.group(1)
        num_converted += 1
        return '<%%= %s %%>' % name

    fixed = MATCH_DECLARE_NEW.sub('', template)
    fixed = MATCH_PYTHON_BLOCK.sub(replace_blocks, fixed)
    fixed = MATCH_FORMAT.sub(convert_variables, fixed)

    # more logic...

Adding elements to stored_blocks works fine, but I cannot increase num_converted in the second nested function. I get an exception that says UnboundLocalError: local variable 'num_converted' referenced before assignment.

I know that in 3.x, I could try nonlocal num_converted, but how can I solve the problem in 2.x? I don't want to use a global variable for this.

Answer 1

Problem: This is because Python's scoping rules are demented. The presence of the += assignment operator marks the target, num_converted, as local to the enclosing function's scope, and there is no sound way in Python 2.x to access just one scoping level out from there. Only the global keyword can lift variable references out of the current scope, and it takes you straight to the top.

Fix: Turn num_converted into a single-element array.

num_converted = [0]
def convert_variables(m):
    name = m.group(1)
    num_converted[0] += 1
    return '<%%= %s %%>' % name

Answer 2

(see below for the edited answer)

You can use something like:

def convert_variables(m):
    name = m.group(1)
    convert_variables.num_converted += 1
    return '<%%= %s %%>' % name

convert_variables.num_converted = 0

This way, num_converted works as a C-like "static" variable of the convert_variable method

---

(edited)

def convert_variables(m):
    name = m.group(1)
    convert_variables.num_converted = convert_variables.__dict__.get("num_converted", 0) + 1
    return '<%%= %s %%>' % name

This way, you don't need to initialize the counter in the main procedure.

Answer 3

Using the global keyword is fine. If you write:

num_converted = 0
def convert_variables(m):
    global num_converted
    name = m.group(1)
    num_converted += 1
    return '<%%= %s %%>' % name

... num_converted doesn't become a "global variable" (i.e. it doesn't become visible in any other unexpected places), it just means it can be modified inside convert_variables. That seems to be exactly what you want.

To put it another way, num_converted is already a global variable. All the global num_converted syntax does is tell Python "inside this function, don't create a local num_converted variable, instead, use the existing global one.

Answer 4

What about using a class instance to hold the state? You instantiate a class and pass instance methods to subs and those functions would have a reference to self...

Answer 5

I have couple of remarks.

First, one application for such nested functions comes up when dealing with raw callbacks, as are used in libraries like xml.parsers.expat. (That the library authors chose this approach may be objectionable, but ... there are reasons to use it nonetheless.)

Second: within a class, there are much nicer alternatives to the array (num_converted[0]). I suppose this is what Sebastjan was talking about.

class MainClass:
    _num_converted = 0
    def outer_method( self ):
        def convert_variables(m):
            name = m.group(1)
            self._num_converted += 1
            return '<%%= %s %%>' % name

It's still odd enough to merit a comment in the code... But the variable is at least local to the class.

Answer 6

Modified from: https://stackoverflow.com/a/40690954/819544

You can leverage the inspect module to access the calling scope's globals dict and write into that. That means this trick can even be leveraged to access the calling scope from a nested function defined in an imported submodule.

import inspect 

def get_globals(scope_level=0):
    return dict(inspect.getmembers(inspect.stack()[scope_level][0]))["f_globals"]

num_converted = 0
def foobar():
    get_globals(0)['num_converted'] += 1

foobar()
print(num_converted) 
# 1

Play with the scope_level argument as needed. Setting scope_level=1 works for a function defined in a submodule, scope_level=2 for the inner function defined in a decorator in a submodule, etc.

NB: Just because you can do this, doesn't mean you should.