How to stop all child processes spawnedby a single script using subprocess

Question

I need to run a unittest on part of my package, and the best thing for me to do this is to control launching it and then killing all processes spawned by multiprocessing module.

Here is what I'm talking about:

test.py

import logging
import multiprocessing
import time
import random

log = logging.getLogger(__name__)

def start_consumers(conf, worker_count=5):
    manager = WorkerManager(conf, worker_count)
    manager.start()


class WorkerManager():
    def __init__(self, conf, worker_count=5):
        self.workers = []
        for num in range(worker_count):
            self.workers.append(WorkHandler(conf))

    def start(self):
        for worker in self.workers:
            worker.daemon = True
            worker.start()

        print 'started'
        for worker in self.workers:
            worker.join()

class WorkHandler(multiprocessing.Process):

    def __init__(self, conf, *args, **kwargs):
        super(WorkHandler, self).__init__(*args, **kwargs)
        self.conf = conf
        self.name = str(random.randint(0,100))

    def run(self):
        while True:
            print self.conf['foo'], self.name
            time.sleep(3)

if __name__ == "__main__":
    conf = {'foo': 'bar'}
    start_consumers(conf)

Now if I run this test from linux terminal, I can see print statements, also if i do: ps aux | grep python I see all child processes where spawned:

sergey    4081  0.0  0.1  71684 13164 pts/3    S+   08:58   0:00 python
sergey    4108  0.3  0.0  39092  6472 pts/3    S+   08:59   0:00 python test.py
sergey    4109  0.0  0.0  39092  4576 pts/3    S+   08:59   0:00 python test.py
sergey    4110  0.0  0.0  39092  4568 pts/3    S+   08:59   0:00 python test.py
sergey    4111  0.0  0.0  39092  4576 pts/3    S+   08:59   0:00 python test.py
sergey    4112  0.0  0.0  39092  4584 pts/3    S+   08:59   0:00 python test.py
sergey    4113  0.0  0.0  39092  4580 pts/3    S+   08:59   0:00 python test.py
sergey    4115  0.0  0.0  13588   944 pts/7    S+   08:59   0:00 grep --color=auto python

Now if I'm trying to run the same test.py with subprocess everything works fine up to the point until I need to kill them all:

>>> import subprocess as s
>>> p = s.Popen(['python', 'test.py'], stdout=s.PIPE)

The Question:

What is the most elegant way to do something with this variable p in order to achieve similar behaviour when I press Ctrl+C in terminal?
In other words I want to achive a result similar to pkill -f "test.py"

UPDATE:

p.kill() or p.terminate() doesn't do what I need:

Here is the result after I do eather of them:

sergey    4438  0.0  0.0      0     0 pts/3    Z+   09:16   0:00 [python] <defunct>
sergey    4439  0.0  0.0  39092  4572 pts/3    S+   09:16   0:00 python test.py
sergey    4440  0.0  0.0  39092  4568 pts/3    S+   09:16   0:00 python test.py
sergey    4441  0.0  0.0  39092  4576 pts/3    S+   09:16   0:00 python test.py
sergey    4442  0.0  0.0  39092  4580 pts/3    S+   09:16   0:00 python test.py
sergey    4443  0.0  0.0  39092  4580 pts/3    S+   09:16   0:00 python test.py

score 2 · Answer 1 · edited May 23 '17 at 11:45

2

I believe what you're looking for is the subprocess.terminate() and subprocess.kill() methods of subprocess objects.

You will likely have to combine that with a method registered with atexit that iterates across your list of subprocesses and terminates them. EX:

def terminate_children(children):
    for process in children:
        process.terminate()

...
# Somewhere else in your code
children = [s.Popen(['python', 'test.py'], stdout=s.PIPE) for i in range(number_processes)] # Spools up number_processes child processes
atexit.register(terminate_children, children) # where children is a list of subprocesses

This will terminate all child processes gracefully when you terminate the parent process gracefully.

If you're trying to kill the processes from a completely separate script with no direct reference to the child code, see here. You'll basically want to look at the python os process management methods

edited May 23 '17 at 11:45

Community

1
1

answered Apr 24 '14 at 13:15

aruisdante

8,875
2
30
37

Not quiet sure I understand where it needs to go? The goal is to not change the source code and to create some kind of a clean up process, but be able to terminate them all from a separate code. – Vor Apr 24 '14 at 13:23
I assume you're spooling up you subprocesses somewhere correct? The ``atexit`` code would go wherever that's happening. Simply store a list of the ``p`` objects you're creating, and pass them as the children parameter in my example. I'll edit it to make that more clear. – aruisdante Apr 24 '14 at 13:27
No it's not what I want, your code will spwn `N` number of processes and each of them will spanw `N` number of subprocesses – Vor Apr 24 '14 at 13:35
See the link I posted at the end. If neither of those are still what you want, you need to make your question clearer. – aruisdante Apr 24 '14 at 13:37
I successfuly used your approach cleaning up too many communicating subprocesseses which causes CPU overload and hangs MAC OS. – Gürol Canbek Dec 11 '16 at 21:23

score 1 · Answer 2 · edited May 23 '17 at 12:08

1

You could try to create a new session and kill all descendant processes using os.killpg():

import os
import signal
from subprocess import Popen

p = Popen('python test.py', shell=True, preexec_fn=os.setsid)
# ... later
os.killpg(p.pid, signal.SIGTERM) # send signal to the process group

See How to terminate a python subprocess launched with shell=True.

edited May 23 '17 at 12:08

Community

1
1

answered Apr 24 '14 at 21:01

jfs

399,953
195
994
1,670

How to stop all child processes spawnedby a single script using subprocess

test.py

The Question:

UPDATE:

2 Answers2