2

Say, I have the following loops:

for i in range(50):
   for j in range(50):
      for k in range(50):
         for l in range(50):
             some_function(i,j,k,l)

some_function happens to be a web scraper for a lot of small files, so it makes sense to parallelize this.

But as far as I can tell, concurrent.futures only accepts one iterator, so I'm not sure how to approach this. What I could do is express (two of) them as a single iterator like:

def in_between_function(a):
    x = math.floor(a/5)
    y = a % y
    some_function(x,y)

with concurrent.futures.ThreadPoolExecutor(max_workers=100) as executor:
   future = executor.map(in_between_function, range(50*50))

That doesn't look too bad but I want to do it properly; if I extend this to more iterators, negative numbers or if the iterator is not linear then this will be a nightmare to maintain.

azro
  • 53,056
  • 7
  • 34
  • 70
Kvothe
  • 386
  • 2
  • 8
  • Does this answer your question? [Pass multiple parameters to concurrent.futures.Executor.map?](https://stackoverflow.com/questions/6785226/pass-multiple-parameters-to-concurrent-futures-executor-map) – Adam.Er8 Nov 10 '19 at 10:34
  • 1
    and to generate the multi-variable range, use [`itertools.product`](https://stackoverflow.com/questions/6785226/pass-multiple-parameters-to-concurrent-futures-executor-map) with `repeat=4` – Adam.Er8 Nov 10 '19 at 10:36
  • I would like to avoid generating a list of (potentially) millions of tuples if possible. Should I? – Kvothe Nov 10 '19 at 11:01
  • 1
    `product` returns a generator, it doesn't create a massive list :) – Adam.Er8 Nov 10 '19 at 11:46

0 Answers0