python asyncio add_done_callback with async def

Question

I have 2 functions: The first one, def_a, is an asynchronous function and the second one is def_b which is a regular function and called with the result of def_a as a callback with the add_done_callback function.

My code looks like this:

import asyncio

def def_b(result):
    next_number = result.result()
    # some work on the next_number
    print(next_number + 1)

async def def_a(number):
    await some_async_work(number)
    return number + 1

loop = asyncio.get_event_loop()
task = asyncio.ensure_future(def_a(1))
task.add_done_callback(def_b)
response = loop.run_until_complete(task)
loop.close()

And it's work perfectly.

The problem began when also the second function, def_b, became asynchronous. Now it looks like this:

async def def_b(result):
    next_number = result.result()
    # some asynchronous work on the next_number
    print(next_number + 1)

But now I can not provide it to the add_done_callback function, because it's not a regular function.

My question is- Is it possible and how can I provide def_b to the add_done_callback function if def_b is asynchronous?

score 34 · Accepted Answer · edited Jan 12 '22 at 15:06

34

add_done_callback is considered a "low level" interface. When working with coroutines, you can chain them in many ways, for example:

import asyncio


async def my_callback(result):
    print("my_callback got:", result)
    return "My return value is ignored"


async def coro(number):
    await asyncio.sleep(number)
    return number + 1


async def add_success_callback(fut, callback):
    result = await fut
    await callback(result)
    return result


loop = asyncio.get_event_loop()
task = asyncio.ensure_future(coro(1))
task = add_success_callback(task, my_callback)
response = loop.run_until_complete(task)
print("response:", response)
loop.close()

Keep in mind add_done_callback will still call the callback if your future raises an exception (but calling result.result() will raise it).

edited Jan 12 '22 at 15:06

MisterMiyagi

44,374
10
104
119

answered Jun 03 '17 at 17:53

Udi

29,222
9
96
129

thank you for sharing, out of curiousity, what is the benefit of using ensure future over using await my_coro within my_callback – laycat Nov 09 '17 at 22:47
1

`ensure_future(coro)` just queues `coro()` for later execution and does not immediately start `coro()`. – Udi Nov 10 '17 at 00:32
10

asyncio.create_task() should be used used instead of asyncio.ensure_future() in Python 3.7+ https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task – kontulai Nov 15 '18 at 15:48
Why this is the right answer. I thought use `asyncio.ensure_future` in callback function is the right one. – BAKE ZQ Oct 18 '19 at 12:41

score 5 · Answer 2 · edited Jan 13 '21 at 02:50

This only works for one future job, if you have multiple async jobs, they will blocks each other, a better way is using asyncio.as_completed() to iterate future list:

import asyncio

async def __after_done_callback(future_result):
    # await for something...
    pass

async def __future_job(number):
    await some_async_work(number)
    return number + 1

loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(__future_job(x)) for x in range(100)]  # create 100 future jobs

for f in asyncio.as_completed(tasks):
    result = await f
    await __after_done_callback(result)

loop.close()

score 0 · Answer 3 · answered Jul 20 '21 at 17:38

You can try the aiodag library. It's a very lightweight wrapper around asyncio that abstracts away some of the async plumbing that you usually have to think about. From this example you won't be able to tell that things are running asynchronously since it's just 1 task that depends on another, but it is all running async.

import asyncio
from aiodag import task

@task
async def def_b(result):
    # some asynchronous work on the next_number
    print(result + 1)

@task
async def def_a(number):
    await asyncio.sleep(number)
    return number + 1

async def main():
    a = def_a(1)
    b = def_b(a)  # this makes task b depend on task a
    return await b

loop = asyncio.get_event_loop()
asyncio.set_event_loop(loop)
response = loop.run_until_complete(main())

python asyncio add_done_callback with async def

3 Answers3

Linked