0

I found the following question for converting sync function to async function. How can I wrap a synchronous function in an async coroutine?.

from functools import wraps, partial


def wrap(func):
    @wraps(func)
    async def run(*args, loop=None, executor=None, **kwargs):
        if loop is None:
            loop = asyncio.get_event_loop()
        pfunc = partial(func, *args, **kwargs)
        return await loop.run_in_executor(executor, pfunc)
    return run

@wrap
def run_async(o):
    s3 = boto3.client('s3')
    s3.put_object(Body=json.dumps(o), ....)
    return o['id']

tasks = [run_async(x) for x in range(500)]
result = asyncio.gather(*tasks)

Will the 500 s3.put_object() run in parallel? Or they cannot be run in parallel due to GIL?

(BTW, I know there is a third party library aioboto3. Will there be run parallel if using aioboto3 without wrapper?)

I did a test. It ran much faster with the wrapper. Why if ran faster even all the code are synchronous?

ca9163d9
  • 27,283
  • 64
  • 210
  • 413
  • I don't think your code works as you expect. You don't actually `await` the `result`. You mix sync with async code to use threads anyway so why not just use [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) instead? – HTF May 18 '22 at 18:47
  • It does improve the speed 5x after using the wrapper. I will look at concurrent.futures. – ca9163d9 May 18 '22 at 18:51

0 Answers0