In Python try until no error

Question

I have a piece of code in Python that seems to cause an error probabilistically because it is accessing a server and sometimes that server has a 500 internal server error. I want to keep trying until I do not get the error. My solution was:

while True:
    try:
        #code with possible error
    except:
         continue
    else:
         #the rest of the code
         break

This seems like a hack to me. Is there a more Pythonic way to do this?

Answer 1

It won't get much cleaner. This is not a very clean thing to do. At best (which would be more readable anyway, since the condition for the break is up there with the while), you could create a variable result = None and loop while it is None. You should also adjust the variables and you can replace continue with the semantically perhaps correct pass (you don't care if an error occurs, you just want to ignore it) and drop the break - this also gets the rest of the code, which only executes once, out of the loop. Also note that bare except: clauses are evil for reasons given in the documentation.

Example incorporating all of the above:

result = None
while result is None:
    try:
        # connect
        result = get_data(...)
    except:
         pass
# other code that uses result but is not involved in getting it

Answer 2

Here is one that hard fails after 4 attempts, and waits 2 seconds between attempts. Change as you wish to get what you want form this one:

from time import sleep

for x in range(0, 4):  # try 4 times
    try:
        # msg.send()
        # put your logic here
        str_error = None
    except Exception as str_error:
        pass

    if str_error:
        sleep(2)  # wait for 2 seconds before trying to fetch the data again
    else:
        break

Here is an example with backoff:

from time import sleep

sleep_time = 2
num_retries = 4
for x in range(0, num_retries):  
    try:
        # put your logic here
        str_error = None
    except Exception as e:
        str_error = str(e)

    if str_error:
        sleep(sleep_time)  # wait before trying to fetch the data again
        sleep_time *= 2  # Implement your backoff algorithm here i.e. exponential backoff
    else:
        break

Answer 3

Maybe something like this:

connected = False

while not connected:
    try:
        try_connect()
        connected = True
    except ...:
        pass

Answer 4

When retrying due to error, you should always:

• implement a retry limit, or you may get blocked on an infinite loop
• implement a delay, or you'll hammer resources too hard, such as your CPU or the already distressed remote server

A simple generic way to solve this problem while covering those concerns would be to use the backoff library. A basic example:

import backoff

@backoff.on_exception(
    backoff.expo,
    MyException,
    max_tries=5
)
def make_request(self, data):
    # do the request

This code wraps make_request with a decorator which implements the retry logic. We retry whenever our specific error MyException occurs, with a limit of 5 retries. Exponential backoff is a good idea in this context to help minimize the additional burden our retries place on the remote server.

Answer 5

The itertools.iter_except recipes encapsulates this idea of "calling a function repeatedly until an exception is raised". It is similar to the accepted answer, but the recipe gives an iterator instead.

From the recipes:

def iter_except(func, exception, first=None):
    """ Call a function repeatedly until an exception is raised."""
    try:
        if first is not None:
            yield first()            # For database APIs needing an initial cast to db.first()
        while True:
            yield func()
    except exception:
        pass

You can certainly implement the latter code directly. For convenience, I use a separate library, more_itertools, that implements this recipe for us (optional).

Code

import more_itertools as mit

list(mit.iter_except([0, 1, 2].pop, IndexError))
# [2, 1, 0]

---

Details

Here the pop method (or given function) is called for every iteration of the list object until an IndexError is raised.

For your case, given some connect_function and expected error, you can make an iterator that calls the function repeatedly until an exception is raised, e.g.

mit.iter_except(connect_function, ConnectionError)

At this point, treat it as any other iterator by looping over it or calling next().

Answer 6

Here's an utility function that I wrote to wrap the retry until success into a neater package. It uses the same basic structure, but prevents repetition. It could be modified to catch and rethrow the exception on the final try relatively easily.

def try_until(func, max_tries, sleep_time):
    for _ in range(0,max_tries):
        try:
            return func()
        except:
            sleep(sleep_time)
    raise WellNamedException()
    #could be 'return sensibleDefaultValue'

Can then be called like this

result = try_until(my_function, 100, 1000)

If you need to pass arguments to my_function, you can either do this by having try_until forward the arguments, or by wrapping it in a no argument lambda:

result = try_until(lambda : my_function(x,y,z), 100, 1000)

Answer 7

Maybe decorator based? You can pass as decorator arguments list of exceptions on which we want to retry and/or number of tries.

def retry(exceptions=None, tries=None):
    if exceptions:
        exceptions = tuple(exceptions)
    def wrapper(fun):
        def retry_calls(*args, **kwargs):
            if tries:
                for _ in xrange(tries):
                    try:
                        fun(*args, **kwargs)
                    except exceptions:
                        pass
                    else:
                        break
            else:
                while True:
                    try:
                        fun(*args, **kwargs)
                    except exceptions:
                        pass
                    else:
                        break
        return retry_calls
    return wrapper


from random import randint

@retry([NameError, ValueError])
def foo():
    if randint(0, 1):
        raise NameError('FAIL!')
    print 'Success'

@retry([ValueError], 2)
def bar():
    if randint(0, 1):
        raise ValueError('FAIL!')
    print 'Success'

@retry([ValueError], 2)
def baz():
    while True:
        raise ValueError('FAIL!')

foo()
bar()
baz()

of course the 'try' part should be moved to another funcion becouse we using it in both loops but it's just example;)

Answer 8

Like most of the others, I'd recommend trying a finite number of times and sleeping between attempts. This way, you don't find yourself in an infinite loop in case something were to actually happen to the remote server.

I'd also recommend continuing only when you get the specific exception you're expecting. This way, you can still handle exceptions you might not expect.

from urllib.error import HTTPError
import traceback
from time import sleep


attempts = 10
while attempts > 0:
    try:
        #code with possible error
    except HTTPError:
        attempts -= 1
        sleep(1)
        continue
    except:
        print(traceback.format_exc())

    #the rest of the code
    break

Also, you don't need an else block. Because of the continue in the except block, you skip the rest of the loop until the try block works, the while condition gets satisfied, or an exception other than HTTPError comes up.

Answer 9

what about the retrying library on pypi? I have been using it for a while and it does exactly what I want and more (retry on error, retry when None, retry with timeout). Below is example from their website:

import random
from retrying import retry

@retry
def do_something_unreliable():
    if random.randint(0, 10) > 1:
        raise IOError("Broken sauce, everything is hosed!!!111one")
    else:
        return "Awesome sauce!"

print do_something_unreliable()

Answer 10

e = ''
while e == '':
    try:
        response = ur.urlopen('https://https://raw.githubusercontent.com/MrMe42/Joe-Bot-Home-Assistant/mac/Joe.py')
        e = ' '
    except:
        print('Connection refused. Retrying...')
        time.sleep(1)

This should work. It sets e to '' and the while loop checks to see if it is still ''. If there is an error caught be the try statement, it prints that the connection was refused, waits 1 second and then starts over. It will keep going until there is no error in try, which then sets e to ' ', which kills the while loop.

Answer 11

Im attempting this now, this is what i came up with;

    placeholder = 1
    while placeholder is not None:
        try:
            #Code
            placeholder = None
        except Exception as e:
            print(str(datetime.time(datetime.now()))[:8] + str(e)) #To log the errors
            placeholder = e
            time.sleep(0.5)
            continue

Answer 12

Here is a short piece of code I use to capture the error as a string. Will retry till it succeeds. This catches all exceptions but you can change this as you wish.

start = 0
str_error = "Not executed yet."
while str_error:
    try:
        # replace line below with your logic , i.e. time out, max attempts
        start = raw_input("enter a number, 0 for fail, last was {0}: ".format(start))
        new_val = 5/int(start)
        str_error=None
    except Exception as str_error:
         pass

WARNING: This code will be stuck in a forever loop until no exception occurs. This is just a simple example and MIGHT require you to break out of the loop sooner or sleep between retries.