Python on Windows - how to wait for multiple child processes?

Question

How to wait for multiple child processes in Python on Windows, without active wait (polling)? Something like this almost works for me:

proc1 = subprocess.Popen(['python','mytest.py'])
proc2 = subprocess.Popen(['python','mytest.py'])    
proc1.wait()
print "1 finished"
proc2.wait()
print "2 finished"

The problem is that when proc2 finishes before proc1, the parent process will still wait for proc1. On Unix one would use waitpid(0) in a loop to get the child processes' return codes as they finish - how to achieve something like this in Python on Windows?

Answer 1

It might seem overkill, but, here it goes:

import Queue, thread, subprocess

results= Queue.Queue()
def process_waiter(popen, description, que):
    try: popen.wait()
    finally: que.put( (description, popen.returncode) )
process_count= 0

proc1= subprocess.Popen( ['python', 'mytest.py'] )
thread.start_new_thread(process_waiter,
    (proc1, "1 finished", results))
process_count+= 1

proc2= subprocess.Popen( ['python', 'mytest.py'] )
thread.start_new_thread(process_waiter,
    (proc2, "2 finished", results))
process_count+= 1

# etc

while process_count > 0:
    description, rc= results.get()
    print "job", description, "ended with rc =", rc
    process_count-= 1

Answer 2

Building on zseil's answer, you can do this with a mix of subprocess and win32 API calls. I used straight ctypes, because my Python doesn't happen to have win32api installed. I'm just spawning sleep.exe from MSYS here as an example, but clearly you could spawn any process you like. I use OpenProcess() to get a HANDLE from the process' PID, and then WaitForMultipleObjects to wait for any process to finish.

import ctypes, subprocess
from random import randint
SYNCHRONIZE=0x00100000
INFINITE = -1
numprocs = 5
handles = {}

for i in xrange(numprocs):
    sleeptime = randint(5,10)
    p = subprocess.Popen([r"c:\msys\1.0\bin\sleep.exe", str(sleeptime)], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
    h = ctypes.windll.kernel32.OpenProcess(SYNCHRONIZE, False, p.pid)
    handles[h] = p.pid
    print "Spawned Process %d" % p.pid

while len(handles) > 0:
    print "Waiting for %d children..." % len(handles)
    arrtype = ctypes.c_long * len(handles)
    handle_array = arrtype(*handles.keys())
    ret = ctypes.windll.kernel32.WaitForMultipleObjects(len(handle_array), handle_array, False, INFINITE)
    h = handle_array[ret]
    ctypes.windll.kernel32.CloseHandle(h)
    print "Process %d done" % handles[h]
    del handles[h]
print "All done!"

Answer 3

Twisted has an asynchronous process-spawning API which works on Windows. There are actually several different implementations, many of which are not so great, but you can switch between them without changing your code.

Answer 4

Twisted on Windows will perform an active wait under the covers. If you don't want to use threads, you will have to use the win32 API to avoid polling. Something like this:

import win32process
import win32event

# Note: CreateProcess() args are somewhat cryptic, look them up on MSDN
proc1, thread1, pid1, tid1 = win32process.CreateProcess(...)
proc2, thread2, pid2, tid2 = win32process.CreateProcess(...)
thread1.close()
thread2.close()

processes = {proc1: "proc1", proc2: "proc2"}

while processes:
    handles = processes.keys()
    # Note: WaitForMultipleObjects() supports at most 64 processes at a time
    index = win32event.WaitForMultipleObjects(handles, False, win32event.INFINITE)
    finished = handles[index]
    exitcode = win32process.GetExitCodeProcess(finished)
    procname = processes.pop(finished)
    finished.close()
    print "Subprocess %s finished with exit code %d" % (procname, exitcode)

Answer 5

You can use psutil:

>>> import subprocess
>>> import psutil
>>> 
>>> proc1 = subprocess.Popen(['python','mytest.py'])
>>> proc2 = subprocess.Popen(['python','mytest.py'])    
>>> ls = [psutil.Process(proc1.pid), psutil.Process(proc2.pid)]
>>>
>>> gone, alive = psutil.wait_procs(ls, timeout=3)

'gone' and 'alive' are lists indicating which processes are gone and which ones are still alive.

Optionally you can specify a callback which gets invoked every time one of the watched processes terminates:

>>> def on_terminate(proc):
...     print "%s terminated" % proc
...
>>> gone, alive = psutil.wait_procs(ls, timeout=3, callback=on_terminate)

Answer 6

you can use psutil

import psutil

with psutil.Popen(["python", "mytest.py"]) as proc1, psutil.Popen(
    ["python", "mytest.py"]
) as proc2:
    gone, alive = psutil.wait_procs([proc1, proc2], timeout=3)

'gone' and 'alive' are lists indicating which processes are gone and which ones are still alive.

Optionally you can specify a callback which gets invoked every time one of the watched processes terminates:

def on_terminate(proc):
    print "%s terminated" % proc

gone, alive = psutil.wait_procs(ls, timeout=3, callback=on_terminate)