Implement coroutine in terms of async def

Question

The following example defines a coroutine multiply, which waits for a value, multiplies it by a constant factor, and prints the result. Another function, product_table, uses multiply to produce product tables.

def multiply(factor):
    print(f"product table for {factor}")
    while True:
        sent_value = yield
        result = factor * sent_value
        print(f"{factor} x {sent_value} = {result}")

def product_table(coro):
    coro.send(None) # start coroutine
    for value in range(1, 11):
        coro.send(value)

product_table(multiply(3))

running the example produces:

product table for 3
3 x 1 = 3
3 x 2 = 6
3 x 3 = 9
3 x 4 = 12
3 x 5 = 15
3 x 6 = 18
3 x 7 = 21
3 x 8 = 24
3 x 9 = 27
3 x 10 = 30

I am trying to implement the exact same example in terms of async def and await, but I'm getting nowhere. My original, incorrect expectation was that the following coroutine function using await was equivalent to the coroutine relying on yield:

async def multiply(factor):
    print(f"product table for {factor}")
    while True:
        await sent_value
        result = factor * sent_value
        print(f"{factor} x {sent_value} = {result}")

It may sound stupid that I expected this to work, but to my dull brain it reads "wait for the sent value". This is not the case -- I get NameError: name 'sent_value' is not defined.

So far, my limited understanding is that you can define coroutine functions in terms of yield (as in the first, working example), or in terms of async def and await, as I attempted in the second example. This does not seem to be the case.

My concrete questions are:

1. How would you implement the first example in terms of async def?
2. Does it even make sense to implement in terms of async def?

I hate that all the freaking examples I've found are in terms of fake pipelines that use time.sleep(0.1) everywhere. I'm trying to make a blog post with examples that use more concrete (if also trivial) pipelines.

[removed erroneous edit]

Answer 1

await is the same thing as yield from, not yield; it delegates control of the generator to a next one.

You are not delegating control, so you would not use async def and await; you are merely sending data across to another generator. If an exception was being sent (with generator.throw()), this would not be passed on to your multiply() generator.

The point of async def is to create awaitable coroutines, and await only works with awaitables. The goal is that you can chain these together and still efficiently get information back to the event loop that drives them. yield is not used (or even legal) in an awaitable coroutine.

You may want to read this excellent blog post by Brett Cannon, a Python core developer, on how async def, await and yield from and @asyncio.coroutine tie together: How the heck does async/await work in Python 3.5?

You can combine a generator with a coroutine; as of Python 3.6 (which implemented PEP 525 - Asynchronous Generators:

async def multiply(factor):
    print(f"product table for {factor}")
    while True:
        sent_value = yield
        result = factor * sent_value
        print(f"{factor} x {sent_value} = {result}")

Note that I didn't replace the yield here. You'd use it like this:

async def product_table(coro):
    await coro.send(None) # start coroutine
    for value in range(1, 11):
        await coro.send(value)

There is little point in making this a coroutine, you are not using any awaitables in the product_table() generator. But say you wanted to send this data to a network socket instead:

async def multiply(factor):
    await socket.send(f"product table for {factor}")
    while True:
        sent_value = yield
        result = factor * sent_value
        await socket.send(f"{factor} x {sent_value} = {result}")

Now you have something that an event loop could manage for you, running multiple such tasks in parallel, sending multiplication tables to multiple clients as they are ready to receive the data.

Answer 2

Does it even make sense to implement in terms of async def?

No, I think it's not.

It happened that asyncio module was implemented using generators, but, theoretically, it could've been implemented without generators at all. There's no point of trying to use asyncio for generator specific job. Just imagine that asyncio coroutines and generator coroutines - are different things.

Roughly saying, you should use asyncio only when you need to do some I/O operations parallely (like downloading multiple urls) and for nothing else. More about it.

Answer 3

When you use async def you are creating a new type of first-class object, a coroutine object. These have to be driven by something, an asynchronous event loop. The await needs to call a coroutine, where it may be suspended. These are usually used for IO.

David Beazley wrote a nice framework for this, called curio. You can see a basic example echo server.