Handling exceptions inside context managers

Question

I have some code where I try to reach a resource but sometimes it is unavailable, and results in an exception. I tried to implement a retry engine using context managers, but I can't handle the exception raised by the caller inside the __enter__ context for my context manager.

class retry(object):
    def __init__(self, retries=0):
        self.retries = retries
        self.attempts = 0
    def __enter__(self):
        for _ in range(self.retries):
            try:
                self.attempts += 1
                return self
            except Exception as e:
                err = e
    def __exit__(self, exc_type, exc_val, traceback):
        print 'Attempts', self.attempts

These are some examples which just raise an exception (which I expected to handle)

>>> with retry(retries=3):
...     print ok
... 
Attempts 1
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
NameError: name 'ok' is not defined
>>> 
>>> with retry(retries=3):
...     open('/file')
... 
Attempts 1
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
IOError: [Errno 2] No such file or directory: '/file'

Is there any way to intercept these exception(s) and handle them inside the context manager?

Answer 1

Quoting __exit__,

If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.

By default, if you don't return a value explicitly from a function, Python will return None, which is a falsy value. In your case, __exit__ returns None and that is why the exeception is allowed to flow past the __exit__.

So, return a truthy value, like this

class retry(object):

    def __init__(self, retries=0):
        ...


    def __enter__(self):
        ...

    def __exit__(self, exc_type, exc_val, traceback):
        print 'Attempts', self.attempts
        print exc_type, exc_val
        return True                                   # or any truthy value

with retry(retries=3):
    print ok

the output will be

Attempts 1
<type 'exceptions.NameError'> name 'ok' is not defined

---

If you want to have the retry functionality, you can implement that with a decorator, like this

def retry(retries=3):
    left = {'retries': retries}

    def decorator(f):
        def inner(*args, **kwargs):
            while left['retries']:
                try:
                    return f(*args, **kwargs)
                except NameError as e:
                    print e
                    left['retries'] -= 1
                    print "Retries Left", left['retries']
            raise Exception("Retried {} times".format(retries))
        return inner
    return decorator


@retry(retries=3)
def func():
    print ok

func()

Answer 2

To deal with an exception in an __enter__ method, the most straightforward (and less surprising) thing to do, would be to wrap the with statement itself in a try-except clause, and simply raise the exception -

But, with blocks are definetelly not designed to work like this - to be, by themselves "retriable" - and there is some misunderstanding here:

def __enter__(self):
    for _ in range(self.retries):
        try:
            self.attempts += 1
            return self
        except Exception as e:
            err = e

Once you return self there, the context were __enter__ runs no longer exists - if an error occurs inside the with block, it will just flow naturally to the __exit__ method. And no, the __exit__ method can not, in anyway, make the execution flow go back to the beginning of the with block.

You are probably wanting something more like this:

class Retrier(object):

    max_retries = 3

    def __init__(self, ...):
         self.retries = 0
         self.acomplished = False

    def __enter__(self):
         return self

    def __exit__(self, exc, value, traceback):
         if not exc:
             self.acomplished = True
             return True
         self.retries += 1
         if self.retries >= self.max_retries:
             return False
         return True

....

x = Retrier()
while not x.acomplished:
    with x:
        ...

Answer 3

I think this one is easy, and other folks seem to be overthinking it. Just put the resource fetching code in __enter__, and try to return, not self, but the resource fetched. In code:

def __init__(self, retries):
    ...
    # for demo, let's add a list to store the exceptions caught as well
    self.errors = []

def __enter__(self):
    for _ in range(self.retries):
        try:
            return resource  # replace this with real code
        except Exception as e:
            self.attempts += 1
            self.errors.append(e)

# this needs to return True to suppress propagation, as others have said
def __exit__(self, exc_type, exc_val, traceback):
    print 'Attempts', self.attempts
    for e in self.errors:
        print e  # as demo, print them out for good measure!
    return True

Now try it:

>>> with retry(retries=3) as resource:
...     # if resource is successfully fetched, you can access it as `resource`;
...     # if fetching failed, `resource` will be None
...     print 'I get', resource
I get None
Attempts 3
name 'resource' is not defined
name 'resource' is not defined
name 'resource' is not defined

Answer 4

I found contextmanager from contextlib useful, hope this may be helpful.

from contextlib import contextmanager

@contextmanager
def handler(*args, **kwargs):
  try:
      # print(*args, **kwargs)
      yield
  except Exception:
      # Handle exception     

Now, to use it,

# Add optional args or kwargs
with handler():
  # Code with probable exception
  print("Hi")

Answer 5

You don't have to implement the retry functionality manually. Take a look at the tenacity library.

Tenacity is a general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.

You can simply add @retry decorator with parameters to your function.

Also,

Tenacity allows you to retry a code block without the need to wraps it in an isolated function. The trick is to combine a for loop and a context manager.