Why doesn't this closure modify the variable in the enclosing scope?

Question

This bit of Python does not work:

def make_incrementer(start):
    def closure():
        # I know I could write 'x = start' and use x - that's not my point though (:
        while True:
            yield start
            start += 1
    return closure

x = make_incrementer(100)
iter = x()
print iter.next()    # Exception: UnboundLocalError: local variable 'start' referenced before assignment

I know how to fix that error, but bear with me:

This code works fine:

def test(start):
    def closure():
        return start
    return closure

x = test(999)
print x()    # prints 999

Why can I read the start variable inside a closure but not write to it? What language rule is causing this handling of the start variable?

Update: I found this SO post relevant (the answer more than the question): Read/Write Python Closures

Your "rebind to a local variable" solution mentioned in a comment will perform better than accessing a container item every time. It's also more Pythonic. See my answer for alternatives, which are also more Pythonic than using a container just for the side-effects. — agf, Sep 24 '11 at 00:13
This does turn out to be an exact duplicate of [Read/Write Python Closures](http://stackoverflow.com/questions/2009402/read-write-python-closures) — agf, Sep 24 '11 at 09:36

Andrew Clark · Answer 1 · 2011-09-23T23:52:05.240

Whenever you assign a variable inside of a function it will be a local variable for that function. The line start += 1 is assigning a new value to start, so start is a local variable. Since a local variable start exists the function will not attempt to look in the global scope for start when you first try to access it, hence the error you are seeing.

In 3.x your code example will work if you use the nonlocal keyword:

def make_incrementer(start):
    def closure():
        nonlocal start
        while True:
            yield start
            start += 1
    return closure

On 2.x you can often get around similar issues by using the global keyword, but that does not work here because start is not a global variable.

In this scenario you can either do something like what you suggested (x = start), or use a mutable variable where you modify and yield an internal value.

def make_incrementer(start):
    start = [start]
    def closure():
        while True:
            yield start[0]
            start[0] += 1
    return closure

So the second bit of code works because it is not assigning to `start` and therefore Python traverses the scopes to the one that I actually want, I take it... — jwd, Sep 23 '11 at 23:50

agf · Accepted Answer · 2011-09-24T00:34:02.180

There are two "better" / more Pythonic ways to do this on Python 2.x than using a container just to get around the lack of a nonlocal keyword.

One you mentioned in a comment in your code -- bind to a local variable. There is another way to do that:

Using a default argument

def make_incrementer(start):
    def closure(start = start):
        while True:
            yield start
            start += 1
    return closure

x = make_incrementer(100)
iter = x()
print iter.next()

This has all the benefits of a local variable without an additional line of code. It also happens on the x = make_incrememter(100) line rather than the iter = x() line, which may or may not matter depending on the situation.

You can also use the "don't actually assign to the referenced variable" method, in a more elegant way than using a container:

Using a function attribute

def make_incrementer(start):
    def closure():
        # You can still do x = closure.start if you want to rebind to local scope
        while True:
            yield closure.start
            closure.start += 1
    closure.start = start
    return closure

x = make_incrementer(100)
iter = x()
print iter.next()

This works in all recent versions of Python and utilizes the fact that in this situation, you already have an object you know the name of you can references attributes on -- there is no need to create a new container for just this purpose.

Excellent explanation except for the `rebinding to a local variable` towards the top -- you are actually just binding. — Ethan Furman, Sep 24 '11 at 00:21
@EthanFurman You're right, I really meant "also bind" not "rebind". — agf, Sep 24 '11 at 00:34

Joe · Answer 3 · 2011-09-24T00:29:03.717

4

Example

def make_incrementer(start):
    def closure():
        # I know I could write 'x = start' and use x - that's not my point though (:
        while True:
            yield start[0]
            start[0] += 1
    return closure

x = make_incrementer([100])
iter = x()
print iter.next()

edited Sep 24 '11 at 00:29

answered Sep 23 '11 at 23:40

Joe

80,724
18
127
145

Hmm, interesting way of getting around the issue; have an upvote. – jwd Sep 23 '11 at 23:52
1

Closure variables are not read-only, or you wouldn't be able to modify them. The quote is also incorrect -- instance `__dict__`s are not closed. – Ethan Furman Sep 24 '11 at 00:03
@Ethan, you're wrong http://stackoverflow.com/questions/141642/what-limitations-have-closures-in-python-compared-to-language-x-closures – Joe Sep 24 '11 at 00:05
Closure variables aren't read-only any more than global variables are (without the `global` statement) -- the problem is Python's assumption about any assigned-to variable, not anything about the closure variables themselves. F.J's answer modifies a closure variable, doesn't it? It just doesn't _rebind a closure name_, which isn't quite the same thing. – agf Sep 24 '11 at 00:09
@agf, in 3.x.x and with a statement declaring it. – Joe Sep 24 '11 at 00:10
You misunderstand. `start[0] += 1` is modifying a closure variable, it's just not _rebinding the name_, which is the operation precluded by Python's assumption about assignment and local variables. If it were truly read-only, you (or F.J) wouldn't be able to do that. – agf Sep 24 '11 at 00:16
@agf, you can't add or remove elements from that array. You can only modify it's contents. – Joe Sep 24 '11 at 00:17
You can add elements, as with `my_closed_var.append(123)`. This seems to be more an issue of scope than of readability/writability... – jwd Sep 24 '11 at 00:20
You can also remove elements -- try `start.append(1)` then `del start[1]` or `start.remove(1)`. – agf Sep 24 '11 at 00:25
1

@agf, I'm wrong. I'm not happy about it, but you guys are correct. – Joe Sep 24 '11 at 00:28
Glad to see you learned something -- correct your answer, and I will happily change my vote. – Ethan Furman Sep 24 '11 at 00:31

Ethan Furman · Answer 4 · 2011-09-24T16:53:32.580

In Python 3.x you can use the nonlocal keyword to rebind names not in the local scope. In 2.x your only options are modifying (or mutating) the closure variables, adding instance variables to the inner function, or (as you don't want to do) creating a local variable...

# modifying  --> call like x = make_incrementer([100])
def make_incrementer(start):
    def closure():
        # I know I could write 'x = start' and use x - that's not my point though (:
        while True:
            yield start[0]
            start[0] += 1
    return closure

# adding instance variables  --> call like x = make_incrementer(100)
def make_incrementer(start):
    def closure():
        while True:
            yield closure.start
            closure.start += 1
    closure.start = start
    return closure

# creating local variable  --> call like x = make_incrementer(100)
def make_incrementer(start):
    def closure(start=start):
        while True:
            yield start
            start += 1
    return closure

@agf: Thanks, fixed my answer. Also, my answer had a better explanation than the others when I wrote it (they have since edited their answers ;) — Ethan Furman, Sep 24 '11 at 00:29

Why doesn't this closure modify the variable in the enclosing scope?

4 Answers4

Using a default argument

Using a function attribute

Linked