Unxepected behavour when access to generator object from list in for loop

Question

I have following code:

l = [(i for j in xrange(2)) for i in xrange(3)]

print list(l[0])

for i in l:
    print list(i)

And following output:

[2, 2]

[]

[<generator object <genexpr> at 0x022B97B0>, <generator object <genexpr> at 0x022B97B0>]

[<generator object <genexpr> at 0x022B9D00>, <generator object <genexpr> at 0x022B9D00>]

First two lines is expected result, but the last two results looks trange for me. I suppose there is no difference between accesses by index l[0] or access each element of list in loop via iterator. Could you please help me?

Answer 1

This problem is caused by a variable name conflict. Variable used by list comprehension shares the global namespace instead of using a isolated one in python 2.x. So, change your variable name from i to something different will give you the right result.

l = [(i for j in xrange(2)) for i in xrange(3)]

print list(l[0])

# Change variable name to `value`
for value in l:
    print list(value)

This will print:

[2, 2]
[]
[2, 2]
[2, 2]

To prove this further, you can do something like this:

l = [(i for j in xrange(2)) for i in xrange(3)]

print list(l[0])

i = 'another value'

for value in l:
    print list(value)

This will give you:

[2, 2]
[]
['another value', 'another value']
['another value', 'another value']

Answer 2

The problem is you've used the variable i in your list comprehension, i will be something totally different from what you expect.

print i
>> 2

So changing it for item will yield the correct result.

for item in enumerate(l):
  list(item)

>> [2, 2]
>> [2, 2]
>> [2, 2]

Answer 3

Your variable i in the list l is not a list of lists but instead it is a reference to a generator object (different in each iteration).

>>> l = [(i for j in xrange(2)) for i in xrange(3)]
>>> for i in l:
...     print i
... 
<generator object <genexpr> at 0xb71ed194>
<generator object <genexpr> at 0xb71ed1bc>
<generator object <genexpr> at 0xb71ed1e4>

PEP 0289 defines about the Generator expressions. It seems quite similar to list comprehension. Quoting directly from PEP 0289:

"Experience with list comprehensions has shown their wide-spread utility throughout Python. However, many of the use cases do not need to have a full list created in memory. Instead, they only need to iterate over the elements one at a time."

Also see this. In your case, try this for instance:

l = [(i for j in xrange(2)) for i in xrange(3)]
k = [[i for j in xrange(2)] for i in xrange(3)]

print list(l[0])

for i in l:
    print list(i)
for i in k:
    print list(i)

In the above code, l is a list of generator objects while k is a list of lists.

Now, consider them one by one. First check for the list l:

>>> l = [(i for j in xrange(2)) for i in xrange(3)]
>>> for i in l:
...     print list(i)
... 
[<generator object <genexpr> at 0xb71c70cc>, <generator object <genexpr> at 0xb71c70cc>]
[<generator object <genexpr> at 0xb71c70a4>, <generator object <genexpr> at 0xb71c70a4>]
[<generator object <genexpr> at 0xb71c71bc>, <generator object <genexpr> at 0xb71c71bc>]
>>> for i in l:
...     print list(i)
... 
[]
[]
[]

You can iterate over a generator object only once. But, in other case, we can use as many times as we want.

>>> k = [[i for j in xrange(2)] for i in xrange(3)]
>>> for i in k:
...     print list(i)
... 
[0, 0]
[1, 1]
[2, 2]
>>> for i in k:
...     print list(i)
... 
[0, 0]
[1, 1]
[2, 2]

But, if you needed a list of tuples then you can use this:

>>> l = [tuple(i for j in xrange(2)) for i in xrange(3)]
>>> l
[(0, 0), (1, 1), (2, 2)]
>>> for i in l:
...     print i
... 
(0, 0)
(1, 1)
(2, 2)
>>> for i in l:
...     print i
... 
(0, 0)
(1, 1)
(2, 2)