0

I started running celery for tasks in a Python/Django web project, hosted on a single VM with 8 cores or CPUs. I need to improve the configuration now - I've made rookie mistakes.

I use supervisor to handle celery workers and beat. In /etc/supervisor/conf.d/, I have two worker-related conf files - celery1.conf and celery1.conf. Should I...

1) Remove one of them? Both spawn different workers. I.e. the former conf file has command=python manage.py celery worker -l info -n celeryworker1. The latter has command=python manage.py celery worker -l info -n celeryworker2. And it's authoritatively stated here to run 1 worker per machine.

2) Tinker with numprocs in the conf? Currently in celery1.conf, I've defined numprocs=2. In celery2.conf, I've defined numprocs=3* (see footer later). At the same time, in /etc/default/celeryd, I have CELERYD_OPTS="--time-limit=300 --concurrency=8". So what's going on? supervisor's numprocs takes precedence over concurrency in celeryd, or what? Should I set numprocs=0?

*total numprocs over both files = 2+3 = 5. This checks out. sudo supervisorctl shows 5 celery worker processes. But in newrelic, I see 45 processes running for celeryd. What the heck?! Even if each proc created by supervisor is actually giving birth to 8 procs (via celeryd), total numprocs x concurrency = 5 x 8 = 40. That's 5 less than the 45 shown by newrelic. Need guidance in righting these wrongs.

Compare screenshots:

5 celery workers as per supervisorctl

vs

45 running celery processes as per newrelic

Community
  • 1
  • 1
Hassan Baig
  • 15,055
  • 27
  • 102
  • 205

1 Answers1

4

it's authoritatively stated here to run 1 worker per machine

Actually, it's advised ("I would suggest") to only run one worker per machine for this given use case.

You may have pretty good reason to do otherwise (for example having different workers for different queues...), and the celery doc states that how many worker / how many processes per worker (concurrency) works best really depends on you tasks, usage, machine and whatnots.

wrt/ numprocs in supervisor conf and concurrency in celery, these are totally unrelated (well, almost...) things. A celery "worker" is actually one main process spawning concurrency children (which are the ones effectively handling your tasks). Supervisor's numprocs tells supervisor how many processes (here: the celery workers) it should launch. So if you have one celery conf with numprocs = 2 and another one with numproc = 3, this means you launch a total of 5 parents worker processes - each of them spawning n subchilds, where - by default - n is your server's cpus count. This means you have a total of 5 + (5*8) = 45 worker subprocesses running.

Wether you actually need that many workers is a question only you can answer ;)

bruno desthuilliers
  • 75,974
  • 6
  • 88
  • 118
  • Thanks, great explanation (and hat!). I guess this brings me to investigating how to reliably tune the number of workers I need. Any direction you can point me into? – Hassan Baig Dec 23 '16 at 16:00
  • Well... First think of what kind of resources your tasks compete for vs how many of that resource is available (given other processes running on the same server), how long your tasks run, how often they are called etc - globally what are your usage patterns. If you have tasks with very different needs / usage patterns, you may want to dispatch them to different queues / different servers (possibly with different resource allocations depending on the queues a server will handle). – bruno desthuilliers Dec 23 '16 at 16:14