Why is this Python generator functioning like this?

Question

I was looking into this article by RealPython about Python generators. It uses a palindrome generator as a exemplary tool to explain the .send, .throw and .close methods. Building a program following their instructions gives the following code:

def infinite_palindromes():
    num = 0
    while True:
        if is_palindrome(num):
            i = (yield num)
            if i is not None:
                num = i
        num += 1


def is_palindrome(num):
    # Skip single-digit inputs
    if num // 10 == 0:
        return False
    temp = num
    reversed_num = 0

    while temp != 0:
        reversed_num = (reversed_num * 10) + (temp % 10)
        temp = temp // 10

    if num == reversed_num:
        return True
    else:
        return False


pal_gen = infinite_palindromes()
for e in pal_gen:
    digits = len(str(e))
    pal_gen.send(10**(digits))

I watched the flow of the program in debugging and what happens is basically, when the function finds 11, the first palindrome, it enters into the for e loop block, the digits variable is defined, and 10 to the power of that variable value is sent back to the generator. 100 is sent back, and since it is different from None, num is equated to i and becomes 100, then it's summed with 1 and becomes 101, the next palindrome. Here is what I don't get: when that number is throw and correctly identified as a palindrome, the code goes back to the for e in pal_gen: line, but it doesn't enter the block. Only the next palindrome, 111 is sent back down and enters the block, where the digits variable is defined again, and so on and so forth, and this patterns repeats, with only every other palindrome entering the block. Why is that?

Can someone, please, explain to me why that happens?

Answer 1

To understand the behaviour, we need to understand how for loops are implemented in Python. This code:

for e in pal_gen:
    digits = len(str(e))
    pal_gen.send(10**(digits))

is basically equivalent to (except that no name _iter is actually generated):

_iter = iter(pal_gen)
while True:
    try:
        e = next(_iter)
        digits = len(str(e))
        pal_gen.send(10**(digits))
    except StopIteration:
        break

However, because pal_gen is a generator object (created by calling the generator function infinite_palindromes), it is already an iterator; calling iter on it returns the same object unchanged. Thus, we effectively have:

while True:
    try:
        e = next(pal_gen)
        digits = len(str(e))
        pal_gen.send(10**(digits))
    except StopIteration:
        break

The other thing to note here is that calling .send on a generator advances the generator, and so does passing it to next (i.e., using it like any other iterator). It's just that the code was written to ignore the value retrieved using .send.

In fact, next on a generator is equivalent to calling .send and passing None (by convention; Python defines the behaviour this way so that the assignment in the generator code, i = (yield num), has something to assign).

So, each time through the loop, two values are sent to the generator:

while True:
    try:
        e = pal_gen.send(None)
        digits = len(str(e))
        pal_gen.send(10**(digits))
    except StopIteration:
        break

The control flow is like so:

• None is sent to the generator. Because it hasn't yielded yet, this value is discarded (Python requires it to be None, since it can't be used by the generator).
• The generator iterates its own loop until 11 is found and yielded. The execution of the generator pauses here: i is not assigned yet.
• The main loop receives 11 as a value for e, computes 100 as the next value to work with, and sends that to the generator.
• The generator receives 100 and assigns it to i, then proceeds with its logic. The next step is if i is not None:, and that is indeed the case - thus, 100 is assigned to num. num is incremented at the end of the loop.
• The loop inside the generator runs into the next iteration, and immediately finds that 101 is a palindrome. Thus, the .send call returns a value of 101, but this isn't assigned anywhere.
• Similarly, in the second iteration of the main loop, 111 will be found and yielded (as the next palindrome after 101) and assigned to e in the first call; then 1000 is sent explicitly, causing 1001 to be yielded and ignored. In the third iteration, 1111 is assigned to e, then 10001 is ignored. In the fourth iteration, 10101 (not 11111! This is the next palindrome after 10001) is assigned, and 100001 is ignored. Etc.

If the code is tested at the REPL, the values returned from the explicit .send will be displayed. This is just a quirk of the REPL.