0

I want to execute N subprocesses as batches with batch size M. I have used this code in a previous SO thread as a starting point:

Python Subprocess Batch Call

def chunks(l, n):
  """Yield successive n-sized chunks from l."""
    for i in xrange(0, len(l), n):
        yield l[i:i+n]

for next_batch in chunks(commands, 5):
    # Start the next few subprocesses
    subps = [subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
             for cmd in next_batch]
    # Wait for these to finish
    for subp in subps:
        subp.wait()

However, how do I handle processes within a batch which do not take the same amount of time to complete (i.e., uneven processes) ? This code still works, but it waits until all processes in a batch complete before executing the next batch.

Is there a way to allow processes in the next batch to start while some processes in the old batch have not yet completed ? While at any given time not exceeding the overall batch size of M

bd3lk
  • 43
  • 6

1 Answers1

0

Use queues : https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue

You push your data to be processed in the queue and then all your workers are pulling from the queue and execute what you need.

MetallimaX
  • 594
  • 4
  • 13