11

I am using Python requests in celery workers to make large number of (~10/sec) API calls(includes GET,POST, PUT, DELETE). Each request takes around 5-10s to complete.

I tried running celery workers in eventlet pool, with 1000 concurrency.

Since requests are blocking process each concurrent connection is waiting on one request.

How do I make requests asynchronous?

Maddy
  • 791
  • 1
  • 8
  • 22

2 Answers2

19

Use eventlet monkey patching to make any pure python library non-blocking.

  • patch single library

    # import requests  # instead do this:
import eventlet
requests = eventlet.import_patched('requests')

    packages erequests and grequests could be stripped down to these two lines.

  • patch everything

    import eventlet
eventlet.monkey_patch()  # must execute as early as possible
...
# everything is non-blocking now:
import requests, amqp, memcache, paramiko, redis

Update: there is known issue with monkey patching requests library. If you get:

ImportError: cannot import name utils

, then modify import line to

requests = eventlet.import_patched('requests.__init__')
maxkoryukov
  • 4,205
  • 5
  • 33
  • 54
temoto
  • 5,394
  • 3
  • 34
  • 50
  • 1
    Perfect ! Only problem was I was unable to do `requests = eventlet.import_patched('requests')` it was giving cannot import utils (`from . import utils`) So I have changed it to `requests = eventlet.import_patched('requests.__init__')` as per https://github.com/eventlet/eventlet/issues/7 thanks again – Maddy Feb 05 '15 at 13:03
  • @Maddy, apart from patching the request library , was anything else required to make requests async and also was there any change in the way the response was collected post request? Post patching can we still do resp = requests.get("http://google.com")? – amrx Mar 30 '17 at 05:19
  • @amrx At the time It worked, but I am sure they must have updated the patching now. – Maddy Mar 31 '17 at 09:10
  • @amrx apart from patching, everything works as expected. – temoto Mar 31 '17 at 09:14
  • @temoto, Do you need to patch? The documentation is not certain about it and even the gevent and eventlet official examples dont patch.https://github.com/celery/celery/blob/master/examples/gevent/tasks.py – amrx Mar 31 '17 at 12:22
  • 1
    @amrx You must either execute `eventlet.monkey_patch()` or use `eventlet.import_patched('module_name')` for every module with network code. Celery does the first option implicitly if you specify eventlet worker type. https://github.com/celery/celery/blob/b5b64945daaba87a7d975b1b4fca08b0948b91bb/celery/__init__.py#L99 – temoto Apr 01 '17 at 16:35
3

from the docs:

there are lots of projects out there that combine Requests with one of Python’s asynchronicity frameworks. Two excellent examples are grequests and requests-futures.

for eventlet specifically you can use erequests.

WeaselFox
  • 7,220
  • 8
  • 44
  • 75