What kind of objects `yield from` can be used with?

Question

Initially (PEP 380), yield from syntax was introduced to be used for delegating to a "subgenerator." Later it was used with now deprecated generator-based coroutines.

I cannot find out what kind of objects yield from can be applied to in general. My first conjecture was that it only requires __iter__ method on the object to return an iterator. Indeed, the following works with Python 3.8:

class C:
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        return iter(range(self.n))

def g(n):
    yield from C(n)

print(tuple(g(3)))

However, it also works with some awaitables, like asyncio.sleep(1), which do not have __iter__ method.

What is the general rule? What determines if an object can be given as an argument to yield from form?

You are in the right path, but perhaps `yield from asyncio.sleep(1)` is confusing you. `yield from` is Python's 3.4 syntax equivalent of Python 3.5 `await`. Check out the documentation for Python 3.4 [asyncio: 18.5.9.3. Concurrency and multithreading](https://docs.python.org/3.4/library/asyncio-dev.html#concurrency-and-multithreading). — felipe, May 20 '20 at 13:48
@Felipe, do you mean that `yield from` has two completely unrelated meanings? Does it have just two of them, or more? In the code example that i gave, you cannot replace `yield from` with `await`, so they are not always equivalent. — Alexey, May 20 '20 at 14:20
@Felipe, i do not see where in the documentation you pointed to my question is answered. — Alexey, May 20 '20 at 14:22
The first paragraph on the link documentation above. "An event loop runs in a thread and executes all callbacks and tasks in the same thread. While a task is running in the event loop, no other task is running in the same thread. But when the task uses `yield from`, the task is suspended and the event loop executes the next task." — felipe, May 20 '20 at 14:35
Asynchronous programming is a different concept all together in Python (different from generators). In today's world, you use the keyword `await` to utilize asynchronous functions, but back before 3.4, you would use `yield from` as opposed to `await` (which is why you might see `yield from` sprinkled around in code that does not seem to have generators). — felipe, May 20 '20 at 14:38
@Felipe, i think you did not understand my question. I am asking what general properties of an object determine if it can be used (for whatever purpose) as an argument to `yield from` (what interface it should implement). — Alexey, May 20 '20 at 14:48
What I am trying to tell you is that this "However, it also works with some awaitables, like `asyncio.sleep(1)`, which do not have `__iter__` method." is not true, and from my understanding, it is the origin of your confusion. `yield from asyncio.sleep(1)` does **not** work in Python 3.8 as [PEP492](https://www.python.org/dev/peps/pep-0492/#rationale-and-goals) removed the use of `yield from` for awaitables, and replaced it with `await`. — felipe, May 20 '20 at 14:53
So to finish the point above, in the current Python 3.8 implementation only generators use the keyword `yield` (whether it is through the implementation of `__iter__`, or just a function that `yield` as oppose to `return`), and thus only Python code that yields from a generator can use `yield from`. :) — felipe, May 20 '20 at 14:55
@Felipe, (1) `yield from asyncio.sleep(1)` *does* work in Python 3.8, (2) whether this use is deprecated is mostly irrelevant to my question, (3) your comments do not address my question. — Alexey, May 20 '20 at 14:59
If you could give an example of when `yield from asyncio.sleep(1)` works outside an asynchronous generator I would appreciate it, and apologize for the inconvenience on my reponses. — felipe, May 20 '20 at 15:02
@Felipe, `@asyncio.coroutine; def c(n): yield from asyncio.sleep(n)`, `asyncio.run(c(1))`. — Alexey, May 20 '20 at 15:04
Oh, I see the confusion. `@asyncio.coroutine` != `async def`. The reason `yield from asyncio.sleep()` works when `@asyncio.coroutine` is written is for backwards compatibility with 3.4, and the use of `yield from` as oppose to `await`. Note that if you replace `@asyncio.coroutine` with `async def` you will receive an error. To answer your question (thanks for clearing things up), `yield from` works with generators, and for backwards compatibility sake only when `@asyncio.coroutine` is present (`yield from` is used in the presence of `@asyncio.coroutine` as `await`). — felipe, May 20 '20 at 15:08
In other words, in 3.4 there was no `await` statement. Core devs first implemented the awaitables with `yield from` (so in essence 3.4 `yield from` == 3.8 `await`). It's a confusing mix of terms, but it's not that `asyncio.sleep()` necessarily has a generator, but that `yield from` was the way to `await` `asyncio.sleep()` in 3.4 (as replicated for backwards compatibility sake when you use `@asyncio.coroutine` as opposed to `async def`). — felipe, May 20 '20 at 15:12

score 3 · Answer 1 · answered May 20 '20 at 14:56

3

You can check how CPython evaluates that statement. From this follows it needs to be either a coroutine or an iterable:

case TARGET(GET_YIELD_FROM_ITER): {
    /* before: [obj]; after [getiter(obj)] */
    PyObject *iterable = TOP();
    PyObject *iter;
    if (PyCoro_CheckExact(iterable)) {
        /* `iterable` is a coroutine */
        if (!(co->co_flags & (CO_COROUTINE | CO_ITERABLE_COROUTINE))) {
            /* and it is used in a 'yield from' expression of a
               regular generator. */
            Py_DECREF(iterable);
            SET_TOP(NULL);
            _PyErr_SetString(tstate, PyExc_TypeError,
                             "cannot 'yield from' a coroutine object "
                             "in a non-coroutine generator");
            goto error;
        }
    }
    else if (!PyGen_CheckExact(iterable)) {
        /* `iterable` is not a generator. */
        iter = PyObject_GetIter(iterable);
        Py_DECREF(iterable);
        SET_TOP(iter);
        if (iter == NULL)
            goto error;
    }
    PREDICT(LOAD_CONST);
    DISPATCH();
}

answered May 20 '20 at 14:56

a_guest

34,165
12
64
118

So, there are really two unrelated meanings... Do you know if it was always so, or if historically all coroutines, including `asyncio.sleep`, had `__iter__` method? – Alexey May 20 '20 at 15:02
2

@Alexey Back then there was no real difference between coroutines and generators. As the [3.4 docs read](https://docs.python.org/3.4/library/asyncio-task.html#coroutines): *"A coroutine is a generator that follows certain conventions."*. No separate coroutine type existed. [PEP 492](https://www.python.org/dev/peps/pep-0492/) which describes the change to `async` and `await` syntax is also informative with respect to the old behavior. [PEP 3156](https://www.python.org/dev/peps/pep-3156/) also provides information on that topic (e.g. search for `yield from` in these docs, it's a long read). – a_guest May 20 '20 at 15:18
I wanted to see how concurrency was implemented with generators, but apparently it is too late, as the implementation changed. I would have to downgrade to Python 3.3, i suppose. – Alexey May 21 '20 at 21:28

What kind of objects `yield from` can be used with?

1 Answers1