0

I am hitting an API using requests library in Django inside a celery task. To be very specific, it fetches some record from database, prepares a json and does a POST request. In the certain case scenario, the call fails with 500 error code. I want to retry the POST request again. What's the best way to go about it and why?

Himanshu Shankar
  • 735
  • 1
  • 6
  • 24

1 Answers1

2

Each Celery job runs in a separate process. In your case, repeating a 500-returned POST request has nothing to do with creating another process. One process will and should be enough of handling such request. So you need to retry the request using urllib.util.retry in the same Celery job and then terminate the job until you get a response with code 200.

cagrias
  • 1,847
  • 3
  • 13
  • 24
  • I have a lot of elements in the queue that require to be posted. What is the disadvantage of using retry in celery? @cagrias – Himanshu Shankar Aug 07 '19 at 11:30
  • 1
    You have only one thing to do: repeat a request until you reach a successful response. As I said, creating a separate celery job will create another process. As making that request from two different processes won't change the result of your request, you do not need to create a separate celery job. – cagrias Aug 07 '19 at 11:35
  • 1
    Celery retry will unnecessary hit your database as many times as it retries - something you really want to avoid... – DejanLekic Aug 07 '19 at 12:13