Python strange behavior with enumerate

Question

I know I'm not supposed to modify the list inside a loop, but just out of curiosity, I would like to know why the number of iterations is different between the following two examples.

Example 1:

x = [1, 2, 3, 4, 5]
for i, s in enumerate(x):
    del x[0]
    print(i, s, x)

Example 2:

x = [1,2,3,4,5]
for i, s in enumerate(x):
    x = [1]
    print(i, s, x)

Example 1 runs only 3 times because when i==3, len(x)==2.

Example 2 runs 5 times even though len(x)==1.

So my question is, does enumerate generate a full list of (index, value) pairs at the beginning of the loop and iterate through it? Or are they generated on each iteration of the loop?

I can't pretend I know enough to answer your question but as to why it behaves differently here is my guess. In the first case you are deleting from the very same list so it makes sense the iteration stops before. In the second case however you are reassigning it. So python probably considers it as a `different` variable and keeps going with the `original` value of x. — bouletta, Mar 07 '17 at 17:35
The same happens without enumerate at all! The `for` loop does not re-evaluate the iterator, so even if you re-assign `x` inside the `for` loop the loop will still use the old value. Obviously if you remove elements from a list, the loop will complete with less iterations. — Bakuriu, Mar 07 '17 at 20:00
enumerate needs to work for unbound sequences, so it wouldn't be possible to generate the pairs up front. If you want to, just use `list(enumerate(...))` — John La Rooy, Mar 08 '17 at 03:52
@jpmc26 indeed there is http://stackoverflow.com/a/986145/3451198 which I did not find yesterday. It does explain the underlying issue very well, however not in the context of a loop. Should this be flagged? I'm rather new at SO and not sure about it. — Wisperwind, Mar 08 '17 at 08:41

score 22 · Answer 1 · answered Mar 07 '17 at 17:36

22

In the first example, you're actually modifying the list you're iterating over.

On the other hand, in the second case, you're only assigning a new object to the name x. The object the loop iterates over does not change, though.

Have a look at http://foobarnbaz.com/2012/07/08/understanding-python-variables/ for a more detailed explanation about names and variables in Python.

answered Mar 07 '17 at 17:36

Wisperwind

963
7
16

Thanks for the answer! I chose Wasi's answer only because it was clearer to understand with the __next__() call. – dbdq Mar 08 '17 at 23:15
@dbdq I think it has more to do with how Python variables work, which is probably why this answer has more upvotes. – JeremiahBarrar May 27 '17 at 14:58
@dbdq I think with my answer it's clear what is going on here – JeremiahBarrar May 27 '17 at 15:11

score 14 · Accepted Answer · edited May 23 '17 at 12:34

enumerate() returns an iterator, or some other object which supports iteration. The __next__() method of the iterator returned by enumerate() returns a tuple containing a count (from start which defaults to 0) and the values obtained from iterating over iterable.

__next__() returns the next item from the container. If there are no further items, raise the StopIteration exception.

Does enumerate() generate a full list of (index, value) pairs at the beginning of the loop and iterates through it? Or are they generated on each iteration of the loop?

So, enumerate() returns an iterator and at every iteration, __next__() checks if there are further items. enumerate() doesn't create a full list at the beginning of the loop.

As, @Wisperwind mentioned, in your second case, you're assigning a new object to the name x. The object, the loop iterates over does not change during the iteration.

So the reason that the original x's value [1, 2,..., 5] does not get garbage collected even after assigning x=[0] is because the iterator is still referring to this list. — dbdq, Mar 08 '17 at 23:26

score 8 · Answer 3 · answered Mar 07 '17 at 18:14

Just a clarification to what Wasi Ahmad and Wisperwind have said. Both state that "you're only assigning a new object to the name x". This might be slightly confusing as it might be interpreted as saying "you're creating a new object ([1]) and storing it to the name x, to which you'd say "Well yah, so why isn't it changing?!" To see what's happening, print out the id of the object

x = [1, 2, 3, 4, 5]
y = x  # To keep a reference to the original list
print id(x), id(y)
for i, v in enumerate(x):
    x = [1]
    print id(x), id(y)
print id(x), id(y)


# output (somewhat contrived as I don't have a python environment set up)
#    X ID            Y ID
10000000000001 10000000000001
10000000000002 10000000000001
10000000000003 10000000000001
10000000000004 10000000000001
10000000000005 10000000000001
10000000000006 10000000000001
10000000000006 10000000000001

You'll notice that the id of x is changing each time through the loop and when you're finished with the loop, x will point to the last modification made in the loop. When you're going through your loop, it is iterating over the original instance of x, regardless of whether you can still reference it.

As you can see, y points to the original x. As you make your iterations through the loop, even though x is changing, y is still pointing to the original x which is still being looped over.

Python is very much a reference based language. You don't assign a value to the name x, you assign a reference to the name x. — Penguin Brian, Mar 08 '17 at 04:46
When I run this code, the id of `x` inside the loop alternates between 139917134004304 and 139917134053248. This is because the new object is being created in the same location as the last-but-one object. However, that wrinkle would make your answer *much* more complicated! — Martin Bonner supports Monica, Mar 08 '17 at 11:02
In addition, if the OP wants to reduce the *original* list that `x` referenced (and that he is looping over), he could write: `del x[1:] : x[0] = 1` — Martin Bonner supports Monica, Mar 08 '17 at 11:06

score 2 · Answer 4 · edited May 23 '17 at 11:47

Others have already pointed out that your second example only changes the value to which x points, but not the list over which you're iterating. This is a perfect example for the difference between ordinary assignment (x = [1]) and slice assignment (x[:] = [1]). The latter modifies the list x points to in-place:

x = [1, 2, 3, 4, 5]
for i, s in enumerate(x):
    x[:] = [1]
    print(i, s, x)

will print

(0, 1, [1])

score 1 · Answer 5 · answered Mar 08 '17 at 09:04

Indeed: Your first snippet modifies the iterated-over list in place; the second points the variable x to a new list, leaving unmodified the list transversed by enumerate(). You can see this in action by going to the following links on www.pythontutor.com, which allow you to single-step over your code and visualize the contents of your variables:

First version (x is modified in place).
Second version (x is redirected to [1]).

To better see what's going on, go here instead to step over the following expanded code:

x = [1,2,3,4,5]
view = enumerate(x)
for i, s in view:
    x = [1]
    print(i, s, x)

JeremiahBarrar · Answer 6 · 2017-05-27T15:17:28.593

x = [1, 2, 3, 4, 5]

The list [1, 2, 3, 4, 5] is 'tagged' with x

for i, s in enumerate(x):

enumerate() attaches another tag, so [1, 2, 3, 4, 5] is now tagged x and y. enumerate() will continue using the y tag, not the x tag.

del x[0]

The list stored in memory is modified, so x and y now both refer to [2, 3, 4, 5]

Alternatively, when you use

x = [1]

A new list [1] is created in memory, and the x tag now points to that. The y tag is still pointing to the original list.

How Python variable work:
http://foobarnbaz.com/2012/07/08/understanding-python-variables/

Python strange behavior with enumerate

6 Answers6