424

Consider the following:

with open(path, mode) as f:
    return [line for line in f if condition]

Will the file be closed properly, or does using return somehow bypass the context manager?

Asclepius
  • 57,944
  • 17
  • 167
  • 143
Lightbreeze
  • 4,624
  • 3
  • 15
  • 10

4 Answers4

380

Yes, it acts like the finally block after a try block, i.e. it always executes (unless the python process terminates in an unusual way of course).

It is also mentioned in one of the examples of PEP-343 which is the specification for the with statement:

with locked(myLock):
    # Code here executes with myLock held.  The lock is
    # guaranteed to be released when the block is left (even
    # if via return or by an uncaught exception).

Something worth mentioning is however, that you cannot easily catch exceptions thrown by the open() call without putting the whole with block inside a try..except block which is usually not what one wants.

ThiefMaster
  • 310,957
  • 84
  • 592
  • 636
  • 11
    `else` could be added to `with` to solve that `try with except` problem. edit: added to language – rplnt Mar 27 '12 at 07:51
  • 7
    I don't know if it's relevant, but to my knowledge `Process.terminate()` is one of the few (the only?) scenario that doesn't guarantee the call of a `finally` statement: *"Note that exit handlers and finally clauses, etc., will not be executed."* – Rik Poggi Mar 27 '12 at 07:55
  • @RikPoggi [`os._exit`](https://docs.python.org/library/os.html#os._exit) is sometimes used - it exits the Python process without calling cleanup handlers. – Asclepius Oct 08 '16 at 06:25
  • 3
    Perhaps taunting the snake a bit, but what if I return a generator expression from within the `with` block, does the guarantee hold for as long as the generator keeps yielding values? for as long as anything references it? I.e. do i need to use `del` or assign a different value to the variable which holds the generator object? – ack Oct 31 '16 at 18:41
  • What happens if something is constructed that depends on the open file - such as `return mmap.mmap(f.fileno(), path.stat().st_size)`? Will the file be closed, breaking the memory map? Or will the fact that the memory map depends on the file descriptor keep the file open for the scope of the returned object? – davidA Apr 12 '18 at 04:45
  • 1
    @davidA After the file is closed, the references are still accessible; however, any attempts to *use* the references to pull/push data to/from the file will give: `ValueError: I/O operation on closed file.`. – RWDJ Sep 05 '19 at 05:24
  • @rplnt agreed. I'd love to see `with` supplemented with `except` and `else` blocks, the same as `try`, but of course without `finally` since it wouldn't serve any purpose. So `with ... except ... else ...` That way you could handle errors that occur when you enter the context handler (e.g. opening a file), or any uncaught errors within the handler (e.g. reading the file), without wrapping the thing in a confusing `try ... except ... else` block. – bob Dec 03 '19 at 20:16
63

Yes.

def example(path, mode):
    with open(path, mode) as f:
        return [line for line in f if condition]

..is pretty much equivalent to:

def example(path, mode):
    f = open(path, mode)

    try:
        return [line for line in f if condition]
    finally:
        f.close()

More accurately, the __exit__ method in a context manager is always called when exiting the block (regardless of exceptions, returns etc). The file object's __exit__ method just calls f.close() (e.g here in CPython)

dbr
  • 165,801
  • 69
  • 278
  • 343
  • 51
    An interesting experiment to show the guarantee you get from the `finally` keywrod is: `def test(): try: return True; finally: return False`. – Ehsan Kia Jul 11 '14 at 19:57
45

Yes. More generally, the __exit__ method of a With Statement Context Manager will indeed be called in the event of a return from inside the context. This can be tested with the following:

class MyResource:
    def __enter__(self):
        print('Entering context.')
        return self

    def __exit__(self, *exc):
        print('EXITING context.')

def fun():
    with MyResource():
        print('Returning inside with-statement.')
        return
    print('Returning outside with-statement.')

fun()

The output is:

Entering context.
Returning inside with-statement.
EXITING context.

The output above confirms that __exit__ was called despite the early return. As such, the context manager is not bypassed.

Asclepius
  • 57,944
  • 17
  • 167
  • 143
10

Yes, but there may be some side effect in other cases, because it may should do something (like flushing buffer) in __exit__ block

import gzip
import io

def test(data):
    out = io.BytesIO()
    with gzip.GzipFile(fileobj=out, mode="wb") as f:
        f.write(data)
        return out.getvalue()

def test1(data):
    out = io.BytesIO()
    with gzip.GzipFile(fileobj=out, mode="wb") as f:
        f.write(data)
    return out.getvalue()

print(test(b"test"), test1(b"test"))

# b'\x1f\x8b\x08\x00\x95\x1b\xb3[\x02\xff' b'\x1f\x8b\x08\x00\x95\x1b\xb3[\x02\xff+I-.\x01\x00\x0c~\x7f\xd8\x04\x00\x00\x00'
virusdefender
  • 505
  • 5
  • 15
  • 1
    In function `test`, `out.getvalue()` happens before `__exit__` is executed, so the result is as expected. – zhy Oct 31 '21 at 04:15