Implementing a retry routine

Question

I had the following idea. Is it possible to implement a retry routine in python? Here is a simple example of what I have done. I would like to have a more flexible solution. Independent of the function. So switch removeFile with any other function and get rid of the while loop in the main.

import os
import time

def removeFile(file):

    try:
        os.remove(file)
        print("removed : "+file)
        return True
    except PermissionError:
        print("could not delete file "+file+" ; will try again")
        return False


if __name__ == "__main__":
    file = "some_path/file.ext"

    sucess = False
    maxCount = 5
    count = 0
    while not sucess:
        sucess = removeFile(file)
        count += 1
        if count == maxCount:
            sucess = True
            print("could not delete file "+file+" ; permission denied.")
        time.sleep(5)

[Here's](https://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/) a nice solution using a decorator. — shmee, Sep 20 '18 at 07:06
Nice thing. I will look into it. Just one little problem decorators only works for methods not for a single line of code, which in my example would be `os.remove(file)` — Sharku, Sep 20 '18 at 07:35
Agreed, but admittedly, you wrap your `os.remove` call in a function as well :) — shmee, Sep 20 '18 at 07:40
Yes you are right. Check out the answer I wrote looks nice I would say. — Sharku, Sep 20 '18 at 07:49
@shmee thanks for the link, that's a nice solution for retrying — Nordle, Sep 20 '18 at 07:52

Sharku · Accepted Answer · 2018-09-20T09:21:09.003

Thanks to @shmee I got a new approach, using a decorator.

def retry(ExceptionToCheck, tries=4, delay=3, backoff=2, logger=None):
    """Retry calling the decorated function using an exponential backoff.

    http://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/
    original from: http://wiki.python.org/moin/PythonDecoratorLibrary#Retry

    :param ExceptionToCheck: the exception to check. may be a tuple of
        exceptions to check
    :type ExceptionToCheck: Exception or tuple
    :param tries: number of times to try (not retry) before giving up
    :type tries: int
    :param delay: initial delay between retries in seconds
    :type delay: int
    :param backoff: backoff multiplier e.g. value of 2 will double the delay
        each retry
    :type backoff: int
    :param logger: logger to use. If None, print
    :type logger: logging.Logger instance
    """
    def deco_retry(f):

        @wraps(f)
        def f_retry(*args, **kwargs):
            mtries, mdelay = tries, delay
            while mtries > 1:
                try:
                    return f(*args, **kwargs)
                except ExceptionToCheck:
                    msg = "%s, Retrying in %d seconds..." % (str(ExceptionToCheck), mdelay)
                    if logger:
                        #logger.exception(msg) # would print stack trace
                        logger.warning(msg)
                    else:
                        print(msg)
                    time.sleep(mdelay)
                    mtries -= 1
                    mdelay *= backoff
            return f(*args, **kwargs)

        return f_retry  # true decorator

    return deco_retry

and now the only thing left to do is decorate our little remove function:

 @retry(PermissionError, tries=5, delay=2,backoff=2)
    def removeFile(f):
        os.remove(f)

You may want to use `logger.exception()` instead so you don't loose the traceback. — bruno desthuilliers, Sep 20 '18 at 08:09

Andrea · Answer 2 · 2018-09-20T08:55:40.100

0

I use this generic template code for re-tries at work to handle all cases and avoid code failures.

N_try = 10

for n_try in range(0,N_try):
    success = None
    try:
        #code execution
        output = None
        output = CodeExecution(input)
    except ex_name_1 as (errno, strerror):
        success = False
        print("EXCEPTION CAUGHT: Try {} of {}:".format(n_try, N_try))
        print("exc_n={}, exc_str={}".format(errno, strerror))
    except ex_name_2 as (errno, strerror):
        success = False
        print("EXCEPTION CAUGHT: Try {} of {}:".format(n_try, N_try))
        print("exc_n={}, exc_str={}".format(errno, strerror))
    except:
        print("UNKNOWN EXCEPTION CAUGHT: Try {} of {}".format(n_try, N_try))
        success = False

    if success is True:
        if CheckOutpup(output) is True:
            success = True
            break
        else:
            success = False
            print("CHECK OUTPUT: Try {n_try} of {N_try} FAILED".format{n_try, N_try})
    else:
        print("EXECUTION: Try {n_try} of {N_try} FAILED".format{n_try, N_try})
        success = False

if success is False:
    print("Failed execution after {N_try} tries".format{N_try})

edited Sep 20 '18 at 08:55

answered Sep 20 '18 at 07:22

Andrea

249
2
7

Ok but you have this template a lot of times in your code or do your wrap that one time around the whole code? – Sharku Sep 20 '18 at 07:34
Only in code parts that I know are critical, such as communication with external devices ( serial-CAN or RF instruments) – Andrea Sep 20 '18 at 07:45
Ok then we are on the same page. Check out the decorator answer I wrote. Might be of use to you, too. – Sharku Sep 20 '18 at 07:50
NEVER use bare except clauses. At least catch `Exception` and LOG THEM. Else you will never know what went wrong nor how to fix it. – bruno desthuilliers Sep 20 '18 at 08:08
You're absolutely right, I forgot this part. The best is to catch all exceptions an after log the type. But in python contrariwise to Matlab is not convenient t catch every exception, you have to know the exception type *before* as explained [here](https://stackoverflow.com/questions/4990718/about-catching-any-exception) – Andrea Sep 20 '18 at 08:47

score -2 · Answer 3 · answered Sep 20 '18 at 07:05

Just put the while loop into the method

import os
import time

def removeFile(file):
    sucess = False
    maxCount = 5
    count = 0

    while not sucess:
        try:
            os.remove(file)
            print("removed : "+file)
            sucess = True
        except PermissionError:
            print("could not delete file "+file+" ; will try again")
            sucess =  False
            count += 1
        if count == maxCount:
            sucess = True
            print("could not delete file "+file+" ; permission denied.")
        time.sleep(5)



if __name__ == "__main__":
    file = "some_path/file.ext"

    removeFile(file)

Implementing a retry routine

3 Answers3

Linked