22

Is it possible to limit the download rate of GET requests using the requests Python library? For instance, with a command like this:

r = requests.get('https://stackoverflow.com/')

...is it possible to limit the download rate? I'm hoping for something similar to this wget command:

wget --limit-rate=20k https://stackoverflow.com/

I know it's possible with urllib2. I'm asking specifically about the requests library.

Community
  • 1
  • 1
Joe Mornin
  • 8,766
  • 18
  • 57
  • 82

2 Answers2

9

There are several approaches to rate limiting; one of them is token bucket, for which you can find a recipe here and another one here.

Usually you would want to do throttling or rate limiting on socket.send() and socket.recv(). You could play with socket-throttle and see if it does what you need.

This is not to be confused with x-ratelimit rate limiting response headers, which are related to a number of requests rather than a download / transfer rate.

Community
  • 1
  • 1
dnozay
  • 23,846
  • 6
  • 82
  • 104
  • 9
    Since the OP asked about doing this with the `requests` library, this doesn't really answer the question... except that you are suggesting he re-write deep portions of `requests` to do what he wants. – Dan H Aug 28 '18 at 20:14
  • @DanH, you are more than welcome to provide a better answer, at time of writing the answer was "no, unless you mess with the internals", which isn't helpful to OP. – dnozay Sep 02 '18 at 17:27
3

No built-in support but, it is possible to use stream api.

>>> import requests
>>> import time
>>> req = requests.request('GET', 'https://httpbin.org/get', stream=True)
>>> for data in req.iter_content(chunk_size=1024):
...     time.sleep(0.001)
...

In advanced usage there is written that its's allow you to retrieve smaller quantities of the response at a time.

In my network the example above (leading to a several GB large file) without sleep had bandwidth 17.4 MB/s and with sleep 1 ms 2.5 MB/s.

frido
  • 789
  • 5
  • 14