How to create a partial list from iterator.combinations()?

Question

I need a list of all combinations of length t for some computation on a GPU. I used X=itertools.combinations(range(n),t) to create an iterator. I need to pass all the combinations to the GPU at once, so I use list(X) to generate all the combinations from the iterator.

For higher values of n and t, I run out of memory. Is there a way to generate multiple smaller lists of all the combinations that can fit in the memory one at a time? For example, if I have n=25 and t=12, instead of generating all 25 choose 12 combinations, can I get a list of 10,000 combinations at a time which I can process on the GPU?

Possible duplicate of [how to split an iterable in constant-size chunks](https://stackoverflow.com/questions/8290397/how-to-split-an-iterable-in-constant-size-chunks) — rdas, Jun 16 '19 at 12:51

score 0 · Accepted Answer · answered Jun 16 '19 at 12:55

from itertools import combinations, count, takewhile

n=25
t=12

chunk_size = 10000

x = combinations(range(n),t)

i = 0
while True:
    l = [*takewhile(lambda _, c=count(): next(c) < chunk_size, x)]

    # your code here, l is a list of length max. 10000 of combinations
    print(l)

    i += 1

    if len(l) < chunk_size:
        break

print('We have done {} iterations.'.format(i))

Prints:

[(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 16), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 17), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 18), (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 19), ...

And at the end:

We have done 520 iterations.

How to create a partial list from iterator.combinations()?

1 Answers1