Chunking a generator

Question

What am I doing wrong here. Trying to get chunks of a generator using islice, but seems to be running infinitely.

from itertools import islice

size = 2

def g():
    for x in range(11):
        print("generating: ", x)
        yield x

while True:
    chunk = islice(g(), size)
    if not chunk:
        break
    print("at chunk")
    for c in chunk:
        print(c)

I am getting an output of, that seems to just loop forever and does not seem to increment:

at chunk
generating:  0
0
generating:  1
1
at chunk
generating:  0
0
generating:  1
1
at chunk
generating:  0
0
generating:  1
1
at chunk

You create a new generator (starting from 0) each iteration of the while loop... — Tomerikoo, Jul 07 '21 at 21:50
All `islice` instances are truthy. You can't tell how many items it will produce without actually iterating over it. — chepner, Jul 07 '21 at 21:50
Compare with the `grouper` recipe in the `itertools` documentation, which uses multiple references to the same underlying iterator to get sequential chunks. — chepner, Jul 07 '21 at 21:52
@Tomerikoo I think the question is more about why this particular attempt failed than how to do it correctly. — chepner, Jul 07 '21 at 21:58

Barmar · Accepted Answer · 2021-07-08T02:10:22.993

3

Each time you call g() you restart the generator from the beginning. You need to assign the result to a variable so it will keep using the same generator.

And as mentioned in a comment, the islice object is always truthy. To tell if you reached the end, check whether the for c in chunk: loop did anything.

from itertools import islice

def g():
    for x in range(11):
        print("generating: ", x)
        yield x

size = 2
gen = g()
while True:
    chunk = islice(gen, size)

    print("at chunk")
    empty = True
    for c in chunk:
        print(c)
        empty = False

    if empty:
        break

edited Jul 08 '21 at 02:10

answered Jul 07 '21 at 21:57

Barmar

741,623
53
500
612

Thank you, this is exactly what I was looking for. I didn't realize I was starting a new generator each time – user1179317 Jul 08 '21 at 04:44

Chunking a generator

1 Answers1