4

This question as been asked with respect to returning from inside the with block, but how about yielding?

Does the block's __exit__ get called on yield and then __enter__ gets called again if the function is next invoked? Or does it wait for the generator to exit the with block (or to return)

as an example :

def tmp_csv_file():
    tmp_path = 'tmp.csv'
    with open(tmp_path, 'w+') as csv_file:
        yield csv_file # will this close the file?
    os.remove(tmp_path)
Community
  • 1
  • 1
Jad S
  • 2,705
  • 6
  • 29
  • 49
  • I would assume not - generators save the state that they yielded at so they can continue from that point. Is there a reason you aren't using the built-in [tempfile](https://docs.python.org/2/library/tempfile.html)? – TemporalWolf Feb 15 '17 at 19:41
  • Couldn't you have just tested this yourself (since you apparently haven't taken the time to read the documentation,either)? – martineau Feb 15 '17 at 19:43
  • 1
    Note: "with" block is called a context manager. :) – Kelvin Feb 15 '17 at 19:44
  • 1
    @TemporalWolf I was ignorant of its existence, thanks for the tip! – Jad S Feb 15 '17 at 20:04
  • @martineau definitely could and would have, however I post on SO to also make the answer accessible to others. Also, MSeifert's answer uncovered some points that I wouldn't tested for, so asking here did pay off :) – Jad S Feb 15 '17 at 20:06

4 Answers4

4

It depends:

  • If your generator function goes out of scope (or is otherwise deleted) the __exit__ is called.

  • If the generator is exhausted the __exit__ is called.

=> As long as the generator is in the with-block and not-exhausted and the variable holding the generator is not deleted the __exit__ is not called.

For example:

class Test(object):
    def __init__(self):
        print('init')

    def __enter__(self):
        print('enter')

    def __exit__(self, *args, **kwargs):
        print('exit')


def funfunc():
    with Test():
        yield 1
        yield 2

Testing it:

>>> a = funfunc()
>>> next(a)  # no exit
init
enter
1
>>> next(a)  # no exit
2
>>> next(a, 0)  # exit because generator leaves the with-context inside the function
exit
0

Or if manually deleted:

>>> a = funfunc()
>>> next(a)
init
enter
1
>>> del a  # exit because the variable holding the suspended generator is deleted.
exit
MSeifert
  • 145,886
  • 38
  • 333
  • 352
  • 1
    accepted this one because the added information on deleting and scope make it the more complete answer – Jad S Feb 15 '17 at 20:02
  • 2
    You don't have to *exhaust* the generator, just advance it out of the `with`. As for deleting it, generators are `close`d on garbage collection, but that doesn't always happen immediately or at all (especially on anything other than CPython), and a buggy generator might not exit the `with` even when `close`d. – user2357112 Feb 15 '17 at 20:06
  • @user2357112 but exhausting is one possibility (because it's equivalent to a `return None`) - even if the context manager isn't exited immediatly it is exited eventually when the object is destroyed (do you have evidence for an implementation that doesn't do that or is it just a guess?). However your last point is interesting: What do you mean by a "buggy" generator that doesn't exit the `with`? – MSeifert Feb 15 '17 at 20:09
  • @MSeifert: [If the generator catches the `GeneratorExit` exception and doesn't exit](http://ideone.com/R0nOVb), `finally` blocks and `__exit__` methods might not run. – user2357112 Feb 15 '17 at 20:17
3

Let's test it empirically.

class MyContextManager:
    def __enter__(self):
        pass
    def __exit__(self, *args):
        print "Context manager is exiting."


def f():
    print("Entered Function.")
    with MyContextManager() as m:
        print("Entered with statement. Yielding...")
        yield m
        print("Yielded. About to exit with statement.")
    print("Now outside of with statement.")

for x in f():
    pass

Output:

C:\Users\Kevin\Desktop>test.py
Entered Function.
Entered with statement. Yielding...
Yielded. About to exit with statement.
Context manager is exiting.
Now outside of with statement.

The "Context manager is exiting" message appears after the "About to exit with statement" message, so we can conclude that the yield statement does not trigger the __exit__ method.

Kevin
  • 74,910
  • 12
  • 133
  • 166
2

yielding inside a with won't trigger __exit__. Control flow inside the generator has to leave the with block for the __exit__ to trigger; suspending the generator doesn't count as leaving the with block. This is similar in spirit to how context switching to another thread won't trigger __exit__ either.

user2357112
  • 260,549
  • 28
  • 431
  • 505
0

In short: no, it will suspend the method from the moment it reaches the yield statement. The remainder is executed in case you ask for the next element.

Given you write result = tmp_csv_file() nothing is done: so not even tmp_path='tmp.csv' is executed.

Now if you call next(result), Python will start to evaluate the function until it hits the first yield statement. So it executes tmp_path = 'tmp.csv', open(..)s the file and __enter__s the environment. It hits the yield statement and thus returns the csv_file. Now you can do whatever you want with that file. The file will remain open (as long as you do not close() it explicitly), and __exit__ will not be called.

If you however call next(result) a second time, Python will continue its quest to find the next yield statement. It will thus __exit__ the with environment, and remove the file (os.remove(tmp_path)). Then it hits the end of the method. This means we are done. And thus next(..) will throw an error that the iterable is exhausted.

Willem Van Onsem
  • 443,496
  • 30
  • 428
  • 555