Celery 4.1.0 and Django 2.0.2 app tasks not discovered

Question

I have found this question and tried all recommendations as well as the accepted answer. Still no luck.

Here is my mysite/taskapp/celery.py:

import os
from celery import Celery
from django.apps import AppConfig
from django.conf import settings

if not settings.configured:
    # set the default Django settings module for the 'celery' program.
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local')  # pragma: no cover


app = Celery('mysite')


class CeleryConfig(AppConfig):
    name = 'mysite.taskapp'
    verbose_name = 'Celery Config'

    def ready(self):
        app.config_from_object('django.conf:settings', namespace='CELERY')
        app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))  # pragma: no cover

Here is my mysite's __init__.py:

from mysite.taskapp.celery import app as celery_app

__all__ = ['celery_app']

Here is an example mysite/myapp/tasks.py:

from celery import shared_task

@shared_task
def mytask():
    print('Do something')

When running the celery worker with celery -A mysite worker -l info here is the output:

celery@tim-office v4.1.0 (latentcall)

Darwin-16.7.0-x86_64-i386-64bit 2018-02-27 21:38:55

[config]
.> app:         mysite:0x1058e0fd0
.> transport:   amqp://guest:**@localhost:5672//
.> results:     disabled://
.> concurrency: 4 (prefork)
.> task events: OFF (enable -E to monitor tasks in this worker)

[queues]
.> celery           exchange=celery(direct) key=celery


[tasks]
. mysite.taskapp.celery.debug_task

Note that the mysite/myapp/tasks.py tasks are not found. I've been spinning my wheels for a full day trying all kinds of things from adding the app to CELERY_IMPORTS in the settings file:

CELERY_IMPORTS = ('myapp.tasks',)

To trying to force them into autodiscover_tasksline in the abovemysite/taskapp/celery.py`:

app.autodiscover_tasks(['myapp',])

Where am I going wrong here? Any help is greatly appreciated.

UPDATE: Possibly worth noting I am using the project structure of cookiecutter-django.

Only when I made each of my apps (in each apps `apps.py` AppConfig class) name like `mysite.myapp` and then included it in `INSTALLED_APPS` as `mysite.myapp.apps.AppConfig` rather than `myapp.apps.AppConfig` did it work. I had to do this for ALL apps (not just the example I showed above). This sucks because I don't like to hardcode the project name into the apps. I prefer to reference it as `myapp.apps.AppConfig` in `INSTALLED_APPS` and import it as `from myapp import something` or `import myapp`. Anyone know if this is an issue with Celery autofinding tasks or something I am doing wrong? — tsantor, Mar 01 '18 at 00:37

score 1 · Answer 1 · answered Feb 27 '18 at 23:45

1

I would comment but don't have 50 reputation.

Do you have logging setup? If so, temporarily disable logging and try to start celery again.

It's possible that there's an uncaught error when Django starts. In my case celery didn't have the correct permissions to a log file.

answered Feb 27 '18 at 23:45

bdoubleu

5,568
2
20
53

Logging enabled or disabled gives same result. It refuses to find the tasks defined in the app. – tsantor Feb 28 '18 at 00:59

himenon · Answer 2 · 2018-03-28T15:52:44.810

I had the same problem, but since I solved it safely, I will respond. My environment is as follows.

Django 4.0
Celery 2.0

First of all, I will show you the sample code. (I am sorry, the language is Japanese.

Sample Code

The main points for making Django recognize tasks are as follows.

In order for Server to recognize Task, app.autodiscover_tasks () is called only once when loading settings.py specified byDJANGO_SETTINGS_MODULE. It does not need to be called on other sub applications.

When starting worker and scheduler, specify DJANGO_SETTINGS_MODULE.

# Server
python3 manage.py runserver 0.0.0.0:8000

# Worker
DJANGO_SETTINGS_MODULE=tm.settings celery -A tm worker

# Scheduler
DJANGO_SETTINGS_MODULE=tm.settings celery -A tm beat --scheduler django_celery_beat.schedulers:DatabaseScheduler

With these two, Django side should recognize the task. Please see Sample Code for details.

References:

Django 4? Current is 2.1 – Routhinator Sep 22 '18 at 00:21 — Routhinator, Sep 22 '18 at 00:21

score 1 · Answer 3 · answered Apr 25 '18 at 19:32

I had the same/a similar issue when using Django 2 and Celery 4.1. It appears that the AppConfig messes up some of the auto-configuration with Celery. I found two solutions:

Don't pass in settings.INSTALLED_APPS to the app.autodiscover_tasks function. This may mess something else up down the road, but haven't run into it yet
Do pass it in, but include an array with just the names of the apps as pointed out here https://stackoverflow.com/a/39129453/502572 (looks you like you mentioned a similar approach as a comment)

Here's my <project_name>/<project_name>/celery.py file:

from __future__ import absolute_import, unicode_literals
import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project_name>.settings')

app = Celery('<project_name>')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

if __name__ == '__main__':
    app.start()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

My __init__.py and <project_name>/<app_name>/tasks.py is setup the same as yours.

Celery 4.1.0 and Django 2.0.2 app tasks not discovered

3 Answers3