How to send a lot of simultaneous requests to FastAPI endpoint?

Question

With a "normal" couroutine like below, the result is that all requests are printed first and then after circa 5 seconds all responses are printed:

import asyncio

async def request():
    print('request')
    await asyncio.sleep(5)
    print('response')

loop = asyncio.get_event_loop()

tasks = [
    loop.create_task(request())
    for i in range(30)
]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

I want to replicate same behavior in FastApi, so I have and endpoint like this:

import asyncio

from fastapi import FastAPI

app = FastAPI()

@app.post("/")
async def root():
    print('request')
    await asyncio.sleep(5)
    return 'OK'

And I am bombing it with multiple requests from the frontend like this:

 const url = 'http://localhost:8000'
 const data = [1, 2, 3]
 const options = {
     method: 'POST',
     headers: new Headers({'content-type': 'application/json'}),
     body: JSON.stringify({data}),
     mode: 'no-cors',
 }

 for (let i=0; i<30; i++) {
     fetch(url, options)
 }

However, in the terminal I can clearly see that the FastAPI accepts only 6 requests at a time, returns responses for them, and then accepts another 6:

request
request
request
request
request
request
←[32mINFO←[0m:     127.0.0.1:63491 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:58337 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:50479 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:60499 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:56990 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:56107 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
request
request
request
request
request
request
←[32mINFO←[0m:     127.0.0.1:58337 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:63491 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:60499 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:56990 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:56107 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m
←[32mINFO←[0m:     127.0.0.1:50479 - "←[1mPOST / HTTP/1.1←[0m" ←[32m200 OK←[0m

etc.

Is this because of some FastAPI/uvicorn setting or limitation?

Can the number be increased, and is it even reasonable to do so?

How are you running your app? Is it just via a bare `uvicorn` command or are you using one of the [pre-made FastAPI docker images](https://fastapi.tiangolo.com/deployment/docker/#official-docker-image-with-gunicorn-uvicorn)? — Gino Mempin, Oct 30 '21 at 06:56

score 1 · Accepted Answer · answered Oct 30 '21 at 09:45

1

You're doing this in your browser, so you're effectively hitting the parallel request limit in your browser. It's not related to the API itself. If you want to test your API's performance, use a tool that has been designed to do that - such as siege, httperf, ab or similar tools.

From the answer documenting the current parallel request limits in browsers:

Firefox 3+: 6
...
Edge:       6
Chrome:     6

answered Oct 30 '21 at 09:45

MatsLindh

49,529
4
53
84

Huh, thanks! I checked my browser limit (Chrome) and it is indeed 6 requests. Assuming no control over browser settings, how would you work around such limit? My guess would be to pass the data needed for such bulk requests from the browser to some backend service that would then query my API. – barciewicz Oct 30 '21 at 11:31
1

There's a few tricks to work around it - for example configuring multiple hosts pointing to the same service (with a wildcard host, for example) since the limitation is per origin. The second would be to instead batch everything in a single request and split it up in your receiving view; performance should be better than for `n` requests. – MatsLindh Oct 30 '21 at 19:54

How to send a lot of simultaneous requests to FastAPI endpoint?

1 Answers1