Why Generators are exhaustive and Lists/Tuples are not?

Question

First of all i have to say i read lot of SO posts before coming to this one because I could not find what I was looking for or maybe I didn't understood. So here it goes

I kind of understand what Iterables and Iterators are. So any container object like Lists/Tuples/Sets which contains items, which you can iterate over are called Iterables. Now to iterate over the Iterables you need Iterators and the way it happens is because of __iter__ method which gives you the Iterator object for the type and then calling the __next__ on the Iterator object to extract the values.

So to make any object iterable you need to define iter and next methods, and i suppose that is true for Lists as well. But here comes the weird part which I discovered recently.

l1 = [1,2,3]
hasattr(l1, "__next__")
Out[42]: False
g = (x for x in range(3))
hasattr(g, "__next__")
Out[44]: True

Now because the lists do support Iterator protocol why the __next__ method is missing from their implementation, and if it indeed is missing then how does iteration for a list work ?

list_iterator = iter(l1)
next(list_iterator)
Out[46]: 1
next(list_iterator)
Out[47]: 2
next(list_iterator)
Out[48]: 3
next(list_iterator)
Traceback (most recent call last):
  File "C:\Users\RJ\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-49-56e733bbb896>", line 1, in <module>
    next(list_iterator)
StopIteration

gen0_iterator = iter(g)
gen_iterator = iter(g)
next(gen_iterator)
Out[57]: 0
next(gen_iterator)
Out[58]: 1
next(gen_iterator)
Out[59]: 2
next(gen_iterator)
Traceback (most recent call last):
  File "C:\Users\RJ\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-60-83622dd5d1b9>", line 1, in <module>
    next(gen_iterator)
StopIteration
gen_iterator1 = iter(g)
next(gen_iterator1)
Traceback (most recent call last):
  File "C:\Users\RJ\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-62-86f9b3cc341f>", line 1, in <module>
    next(gen_iterator1)
StopIteration

I created an iterator for a list and then called next method on it to get the elements and it works.

1. Now if the previous hasattr(a, "__next__") returns a False then how we are able to call next method on the iterator object for a list.

2. Now the original question which made me think all this, no matter how many times i iterate over the list, it doesn't exhaust and calling the iter() gives back a new iterator object everytime, but in case of generator this does not happen, and once the generator has exhausted, no matter how many times you call iter() it will always gives you back the same object which already has raised the StopIteration exception and again this is true because an iterator once raised a StopIteration, it always will, but why it does not happen with lists.

Further this is in sync with what python docs says for conatiner.__ iter__ that container.__iter__ gives you the iterator object for the type and iterator.__ iter__ and iterator.__iter__ gives you the iterator object itself, which is precisely the reason that calling the iter() on generator returns the same object over and over again. But why and more importantly how ?

One more thing to observe here is

isinstance(l1 , collections.Iterator)
Out[65]: False
isinstance(g , collections.Iterator)
Out[66]: True

So this suggests that there is some implementation difference b/w Iterables and Iterators, but i could not find any such details, because both have __iter__ and __next__ methods implemented, so from where does this variation in behavior comes. So is it that __iter__ for iterables returns something different from what is returned by __iter__ of iterables(generators). If some can explain with some examples of __iter__ for Iterables and Iterataors that would be really helpful. Finally some puzzle about yield, since that is the magic word which makes a normal function a generator (so a type of iterator), so what does __iter__ and __next__ of `yield looks like.

I have tried my level best to explain the question, but if still something is missing, please do let me know i will try to clarify my question.

Answer 1

Its a bit different than that. iterables have an __iter__ method that returns an iterator. iterators have a __next__ method (and usually also have __iter__ so that iter() works on them - but that's not required).

Lists are iterable:

>>> l = [1,2,3]
>>> hasattr(l, "__iter__")
True
>>> hasattr(l, "__next__")
False
>>> l_iter = iter(l)
>>> hasattr(l_iter, "__next__")
True
>>> hasattr(l_iter, "__iter__")
True
>>> l_iter == iter(l_iter)
True

And give you new iterators that run through the e each time you use them

>>> list(l)
[1, 2, 3]
>>> list(l)
[1, 2, 3]
>>> l_iter = iter(l)
>>> list(l_iter)
[1, 2, 3]
>>> list(l_iter)
[]
 each time you use them

>>> list(l)
[1, 2, 3]
>>> list(l)
[1, 2, 3]
>>> iter(l) == iter(l)
False

But the list iterator itself is one shot

>>> l_iter = iter(l)
>>> list(l_iter)
[1, 2, 3]
>>> list(l_iter)
[]

The generator is an iterator, not an iterable and is also one shot.

>>> g = (x for x in range(3))
>>> hasattr(g, "__iter__")
True
>>> hasattr(g, "__next__")
True
>>> g == iter(g)
True
>>> 
>>> list(g)
[0, 1, 2]
>>> list(g)
[]