Celery chain breaks if one of the tasks fail

Question

What is the point of Celery chain if the whole chain breaks if one of the tasks fail?!!

I have this Celery chain:

res = chain(workme.s ( y=1111 ), workme2.s( 2222 ), workme3.s( 3333 ),)()

And I made workme2 fails with retries like this:

@celery.task(default_retry_delay=5, max_retries = 10, queue="sure") 
def workme2(x,y):
    # try:      
    try:
        print str(y)
        sleep(2)
        print str(x)
        ## adding any condition that makes the task fail
        if x!=None:
            raise Exception('Aproblem from your workme task')
        print 'This is my username: ' + str(x['user']) + \
               ' And Password: ' + str(x['pas'])        
        return "22xx"
    except Exception, exc:
        workme2.retry(args=[x,y], exc=exc,)

http://stackoverflow.com/questions/11508112/retrying-celery-failed-tasks-that-are-part-of-a-chain — Bernhard Vallant, Jun 25 '13 at 14:05
@BernhardVallant, Hi, I downloaded the latest a couple of days ago, does this mean this patch is not included?? — securecurve, Jun 25 '13 at 15:59
I downloaded the latest version from the master branch on github, do I have to do something else. I just have one question for you .. did you try it yourself?? If you did and worked for you, please tell me yes, and send me the download link from github — securecurve, Jun 26 '13 at 13:00

score 25 · Accepted Answer · answered Sep 11 '15 at 08:44

That is the point.

Forming a chain means your subtasks have some kind of serial dependency: Each one only makes sense if the previous one has been performed. Without this, you would simply use queueing or use a group rather than a chain.

So if one subtask fails (and still fails after attempting all its retries), the chain fails.

I readily admit that the documentation (as of Celery 3.1.18) is far from explicit in this respect, but the name suggests this semantics: "A chain is only as strong as its weakest link."

Good point :) .. and I agree with you that there are some blurry points in the documentation — securecurve, Sep 11 '15 at 08:59

Celery chain breaks if one of the tasks fail

1 Answers1