0

I'd like to start and stop the celery worker processes from inside a Python script. I'm using Redis as the broker and backend.

I issue a Popen command to start the workers from inside the script that schedules tasks for workers:

  # start the celery deamon
  cmd = 'celery worker '
  cmd += '--app intertext.tasks '
  cmd += '--loglevel critical'
  subprocess.Popen(shlex.split(cmd))

After all steps are complete, I want to delete all worker processes. I know I could do something like ps -ef | grep celery | awk '{print $2}' | xargs kill -9 but that won't run on Windows.

What's the best way to kill the processes opened with Popen (or the best way to start and stop the celery deamon from within a Python script)?

duhaime
  • 25,611
  • 17
  • 169
  • 224
  • how about noting down the `pid` of processes that are opened via `Popen`? https://docs.python.org/3/library/subprocess.html#subprocess.Popen.pid – Ali Yılmaz May 12 '18 at 11:14
  • @AliYılmaz how can I make this also work on Windows? I'm reading https://stackoverflow.com/questions/4789837/how-to-terminate-a-python-subprocess-launched-with-shell-true – duhaime May 12 '18 at 11:18
  • hmm I found this: https://stackoverflow.com/a/28609523/7757415 – Ali Yılmaz May 12 '18 at 11:20

2 Answers2

1

I ended up using the following approach:

from psutil import Process
from subprocess import Popen
import shlex

def start_celery():
  '''
  Start the celery deamon.
  @returns:
    subprocess.Popen object that supports pid lookup via
    self.pid
  '''
  cmd = 'celery worker '
  cmd += '--app intertext.tasks '
  cmd += '--loglevel error'
  return subprocess.Popen(shlex.split(cmd))


def terminate_process(pid):
  '''
  Stop a process given its pid or stop all processes if pid
  is a list of identifiers.
  @args:
    int pid: the process identifier of the root process that
      spawned child celery processes
  '''
  process = Process(pid)
  for child in process.children(recursive=True):
    child.kill()
  process.kill()

# main
celery_process = start_celery()

# ...do work... then
terminate_process(celery_process.pid)
duhaime
  • 25,611
  • 17
  • 169
  • 224
0

Firstly, type this script to show all celery workers : 

ps -aux | grep celery

Then kill all celery worker as follow :

pkill -9 -f 'worker_name worker'

I hope this can help you to solve your problem.

Taha Hamedani
  • 105
  • 1
  • 11