2

I have a small API written in Python with Flask. It is a wrapper API around another API (which I don't control). I need to make several requests and I want to make them parallel. The first alternative was to use grequests but inexplicably the uwsgi server hanged and didn't respond. It took a while for it to reload workers after sending HUP signal to the master process. The code I was using is something like:

urls = [
    'http://www.example.com', 
    'http://www.example.net',
]
urls = [grequests.get(x, hooks=dict(response=function_to_do_some_work))
        for url in urls]
grequests.map(urls) # <- here it hangs

I thought it was some problem with gevent/grequests so I reimplemented the solution using multiprocessing.Pool, like this:

urls = [
    'http://www.example.com', 
    'http://www.example.net',
]
pool = multiprocessing.Pool(10)
pool.map(function_to_do_some_work, urls) # <- here it hangs

The problem here is not the code, that runs perfectly in the development environment. The problem is with the uwsgi and Python execution model, or something between those lines. I don't know where to start looking for answers here. The versions of the modules I'm using are:

Python 2.7.5
grequests 0.2.0
Flask 0.10.1
uwsgi 2.0.9

What alternative of parallel execution should I use to perform several tasks at the same time, under this execution environment?

licorna
  • 5,670
  • 6
  • 28
  • 39
  • 3
    Your application container (uWSGI, or whatever you want) need to know if you want to use a specific development paradigm. grequests needs a gevent hub, so uWSGI must be run in gevent mode. For using user-generated threads you need to enable them with enable-threads like explained in the first answer – roberto Feb 07 '15 at 08:02
  • 1
    I was having the same problem using `threading.Thread` + `uwsgi` and adding `enable-threads = true` worked like a charm! +1 – Caumons May 20 '17 at 15:17

1 Answers1

3

Threading needs to be enabled on uwsgi by 

enable-threads = True

Full example of config of nginx, and uwsgi, upstart, virtual-env see

https://stackoverflow.com/a/27221427/567606

Community
  • 1
  • 1
tourdownunder
  • 1,779
  • 4
  • 22
  • 34