Python - What other options are there when requests Retry for rate limit error still gives error?

Question

The API docs say the following:

The overall rate limit is 60 requests per minute. This rate limit applies to any combination of API methods that are called within a trailing 1-minute timeframe

I'm trying the below that gets a response for each page, but I see the following error:

"ok": false, "error": "rate_limited" 429 status code

I feel like the backoff_factor isn't working because I have it set to 1 but I see this error after 18seconds of my code running, shouldn't it run for at least 1 minute?

def get_request():

   session = requests.Session()
   retry = Retry(total=5, connect=7, backoff_factor=1)
   adapter = HTTPAdapter(max_retries=retry)
   session.mount('https://', adapter)

   url = "https://my_url" 

   first_page = session.post(url).json()
   yield first_page
   num_pages = first_page['last_page']
   for page in range(2, num_pages + 1):
       next_page = session.post(url, params={'page': page}).json()
       yield next_page

for page in get_request():
     etc...


   list = []
   for page in range(2, num_pages + 1):
              response = sessions.post(url, params={'page': page}).json()
      r_json = response.json()
      list.append(r_json)

The HTTP rate limit is a server-side configuration and cannot be changed from the client-side. — Artyom Vancyan, Jun 23 '22 at 10:48
@ArtyomVancyan - got it! but what would be the right way to go about working around the rate limit to get the code working? — KristiLuna, Jun 23 '22 at 13:54
@ArtyomVancyan - thanks! Do you know of a good resource to learn more about this and some example maybe? — KristiLuna, Jun 23 '22 at 14:38
https://stackoverflow.com/questions/8287628/proxies-with-python-requests-module — Artyom Vancyan, Jun 23 '22 at 14:40

score 0 · Accepted Answer · answered Jun 23 '22 at 19:10

Why do you think it will run for at least 1 minute? As per the documentation for Retry:

A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for:

{backoff factor} * (2 ** ({number of total retries} - 1)) seconds.

If the backoff_factor is 0.1, then Retry.sleep() will sleep for [0.0s, 0.2s, 0.4s, …] between retries. It will never be longer than backoff_max.

You have the backoff_factor set to 1, with the total retries set to 5. This means that at most there can be 1 original + 5 retry requests sent to the server, if requests receive a 429 error. Going step-by-step, this is how your requests will be made:

Original request sent
Server returned 429 error, first retry made after 0 seconds 1 * (2 ** (1-1))
Server returned 429 error, second retry made after 2 seconds 1 * (2 ** (2-1))
Server returned 429 error, third retry made after 4 seconds 1 * (2 ** (3-1))
Server returned 429 error, fourth retry made after 8 seconds 1 * (2 ** (4-1))
Server returned 429 error, fifth retry made after 16 seconds 1 * (2 ** (5-1))

This gives a total of at least 30 seconds for which your code will wait before raising an error (excluding latencies). It is possible that you made a mistake in calculating how long the your program runs before an exception is raised (i.e. it ran for more than 18s), and it is also possible that the server enforces a lower limit than what is stated, rate-limits can be fickle.

In any case, if you want the program to wait till the rate limit expires (the publicly stated one anyway), then either increase the total retries, or increase the backoff_factor (using the formula) so that the total wait time comes larger than 60.

Reference

Python - What other options are there when requests Retry for rate limit error still gives error?

1 Answers1