Why is Exception output not captured by contextlib.redirect_stderr context manager?

Question

In the following code, I would have expected the exception output and traceback to be written to test.txt file, but that does not happen.

import contextlib

with open("test.txt", "a") as f:
    with contextlib.redirect_stderr(f):
        raise Exception("Hello")

Why is it not working as expected and how can I correctly redirect the exception output to a file?

score 2 · Accepted Answer · answered Sep 28 '21 at 15:40

Because redirecting sys.stderr doesn't catch exceptions in python (it would be bad if it did, right? Because exceptions are only supposed to be caught by try/except blocks.) so the exception handling code interrupts your running code (and the redirect).

Proof:

import contextlib
from sys import stderr

with open("test.txt", "a") as f:
    with contextlib.redirect_stderr(f):
        sys.stderr.write("hello")
        sys.stderr.flush()

writes correctly.

To redirect the output from all exceptions to a file you have two options: catch all exceptions globally in a try/except block and do the writing yourself, or use shell redirection to send the script's stderr to a file, with something like

python my_script.py 2>errors.txt

If you want to do it yourself, see this question for discussion of getting the traceback.

Thanks. I think I can wrap my head around it: it's because the runtime, including the contextlib's stderr redirection is interrupted and stops when the exception arises, the output being then propagated outside the redirection. I went for a try/except block solution (as you suggest) inside a custom context manager — Jean Monet, Sep 30 '21 at 07:38

score 0 · Answer 2 · answered Sep 28 '21 at 16:11

Just to add an alternative implementation I went for.

The following is redirecting the exception output to a logger. In my case the logger was set to stdout (default) so I used contextlib.redirect_stdout to redirect the logger's output to a file, but you could of course directly make the logger write to a file.

import logging
import contextlib
from typing import Iterator


@contextlib.contextmanager
def capture_exception(logger: logging.Logger) -> Iterator[None]:
    """
    Captures exceptions and redirects output to the logger
    >>> import logging
    >>> logger = logging.getLogger()
    >>> with capture_exception(logger=logger):
    >>>     raise Exception("This should be outputed to the logger")
    """
    try:
        # try/except block where exception is captured and logged
        try:
            yield None
        except Exception as e:
            logger.exception(e)
    finally:
        pass

Usage:

logger = logging.getLogger()   # default should be stdout

with open("test.txt", "a") as f:
    with contextlib.redirect_stdout(f), capture_exception(logger):
        # ^-- multiple context managers in one `with` statement
        raise Exception("The output of this exception will appear in the log. Great success.")

Exception output as it appears in the log:

The output of this exception will appear in the log. Great success.
Traceback (most recent call last):
  File "/tmp/ipykernel_9802/2529855525.py", line 52, in capture_exception
    yield None
  File "/tmp/ipykernel_9802/3344146337.py", line 9, in <module>
    raise Exception("The output of this exception will appear in the log. Great success.")
Exception: The output of this exception will appear in the log. Great success.

Why is Exception output not captured by contextlib.redirect_stderr context manager?

2 Answers2