How to await method in loop?

Question

I have the following method that generates data for me:

async def generate_url(self, video_id):
    data = await self.s3.generate_presigned_url(...video_id...)
    return data

def convert_to_json(self, urls):
    ids = [self.generate_url(url) for url in urls]
    ...

How to correctly await generate_url in convert_to_json?

Possible duplicate of [Asynchronous method call in Python?](http://stackoverflow.com/questions/1239035/asynchronous-method-call-in-python) — Neil, Apr 04 '17 at 19:09
assuming `convert_to_json` was async then you would just await for each element of `ids`, I think `async for data in ids:` would work. — Tadhg McDonald-Jensen, Apr 04 '17 at 19:14

score 11 · Accepted Answer · answered Apr 05 '17 at 21:25

You may use a wait wrapper around the list of tasks:

async def convert_to_json(self, urls):
    tasks = [self.generate_url(url) for url in urls]
    await asyncio.wait(tasks)

Or, if you can't mark convert_to_json method as async, wait it synchronously:

import asyncio

def convert_to_json(self, urls):
    loop = asyncio.get_event_loop()
    tasks = [self.generate_url(url) for url in urls]
    loop.run_until_complete(asyncio.wait(tasks))

Also you can try to implement an async iterator and use it with async for syntax, something like this:

class Loader:
    def __init__(self, urls):
        self._urls = iter(urls)

    async def generate_url(self, video_id):
        data = await self.s3.generate_presigned_url(...video_id...)
        return data

    def __aiter__(self):
        return self

    async def __anext__(self):
        try:
            url = next(self._urls)
        except StopIteration:
            raise StopAsyncIteration
        data = await self.generate_url(url)
        return data

async for id in Loader(urls):
    print(id)

score 1 · Answer 2 · edited Apr 05 '17 at 21:25

For me it worked something like this:

import asyncio

class A:
    def __init__(self):
        pass

    async def generate_url(self, video_id):
        data = await self.s3.generate_presigned_url(...video_id...)
        return data

    def add_to_ids(self, id):
        ids.append(id.result())

    def convert_to_json(self, urls):
        loop = asyncio.get_event_loop()

        ids = []
        tasks = []

        for url in urls:
            task = asyncio.ensure_future(self.generate_url(url))
            task.add_done_callback(self.add_to_ids)
            tasks.append(task)
        tasks = asyncio.gather(*tasks)

        loop.run_until_complete(tasks)
        loop.close()

a = A()
a.convert_to_json([1, 2, 3])

Enjoy!

score 1 · Answer 3 · answered Apr 05 '17 at 22:03

1

If you don't use a loop already, try:

loop = asyncio.get_event_loop()
ids = loop.run_until_complete(asyncio.gather(*[self.generate_url(url) for url in urls]))

Or in a coroutine:

ids = await asyncio.gather(*[self.generate_url(url) for url in urls])

answered Apr 05 '17 at 22:03

Udi

29,222
9
96
129

How to await method in loop?

3 Answers3