0

I Want to create a large amount of subprocesses(100+) that will run synchronously through subprocess.call. This will greatly decrease performance and assure me of a crash or two.

My goal is to limit the amount of subprocesses running at the same time while still looping over the main list.

Concept:

for line in list:
    subprocess.call('/root/folder/to/my/script' + line)
    i+=1
    if i > 10: 
        if subprocess.call(stdout == 0):
            create new subprocess

So I want to loop over a list, create 10 subprocess.calls and if one of the subprocess.call is finished running (stops giving output for a x amount of time?) or if one of the subprocess.call is crashing: create a new one.

Here is a more visual explanation of the question:

Running processes:
  [o]    [o]
  [o]    [o]
  [o]    [X] <--Crashed/Finished: Replace with new subprocess.call from list
  [o]    [o]
  [o]    [o]

I Hope I maid my question/problem clear, if I didn't made myself clear enough please tell me.

Nielsv
  • 1
  • 3

1 Answers1

0

Have a look at the multiprocessing module, especially the Pool object.

EvertW
  • 1,160
  • 9
  • 18
  • But multiprocessing is designed for multiprocessing. What I would like to do is run multiple scripts (as a subprocess/threat?) that will run parallel/synchronous. I Would also like to be able to control those subprocesses. – Nielsv Nov 24 '15 at 14:00
  • @Nielsv: `multiprocessing` also provides a thread pool that is a simple way (codewise) to run a fixed number of concurrent subprocesses. See [Python threading multiple bash subprocesses?](http://stackoverflow.com/q/14533458/4279) and if you need to collect the output then see [Python: execute cat subprocess in parallel](http://stackoverflow.com/a/23616229/4279) – jfs Nov 25 '15 at 14:05