Retry function in Python

Question

Some time ago, I was in need of a retry function in R to handle slow servers response. The function would have the following behavior : (try an action (function or method), and if it fails, wait a bit and then retry)x10

I came up with the following :

retry <- function(fun, max_trys = 10, init = 0){
  suppressWarnings(tryCatch({
    Sys.sleep(0.3);
    if(init<max_trys) {fun}
}, error=function(e){retry(fun, max_trys, init = init+1)}))}

It worked well. Now I need the same in Python3, so I tried to make the same code :

import time
def retry_fun(fun, max_trys = 10, init=0):
    try:
        time.sleep(0.3)
        if(init<max_trys):
            fun
    except:
        retry_fun(fun, max_trys, init = init+1)

But when I run it, it's crashing my kernel. As I'm a bit of a beginner in Python, I'm not sure what's causing the crash, and if/how a function can be passed as an argument into another function.

Could you help me out ?

Answer 1

Apart from being able to pass functions and use them by adding () after the name (Python's syntax for invoking calls), you don't need to use recursion; just put it in a loop:

import time
def retry(fun, max_tries=10):
    for i in range(max_tries):
        try:
           time.sleep(0.3) 
           fun()
           break
        except Exception:
            continue

except Exception should be changed to catch a meaningfull exception that the function might raise. Using Exception (as I did in the example) is generally bad practice since it catches a large class of exceptions that you might not of wanted caught.

Apart from that, using a for-loop instead of an explicit third counter and recursion (which leads to a long call-stack for large values) is better.

Answer 2

I know this is an old question. However, I'd like to add the solution that I've cooked up. The best way is to write a retry decorator that will retry when an exception occurs. Moreover, you can also set custom exponential delay. The docstring explains how you can use the decorator. Here you go:

import logging
import time
from functools import partial, wraps


def retry(func=None, exception=Exception, n_tries=5, delay=5, backoff=1, logger=False):
    """Retry decorator with exponential backoff.

    Parameters
    ----------
    func : typing.Callable, optional
        Callable on which the decorator is applied, by default None
    exception : Exception or tuple of Exceptions, optional
        Exception(s) that invoke retry, by default Exception
    n_tries : int, optional
        Number of tries before giving up, by default 5
    delay : int, optional
        Initial delay between retries in seconds, by default 5
    backoff : int, optional
        Backoff multiplier e.g. value of 2 will double the delay, by default 1
    logger : bool, optional
        Option to log or print, by default False

    Returns
    -------
    typing.Callable
        Decorated callable that calls itself when exception(s) occur.

    Examples
    --------
    >>> import random
    >>> @retry(exception=Exception, n_tries=4)
    ... def test_random(text):
    ...    x = random.random()
    ...    if x < 0.5:
    ...        raise Exception("Fail")
    ...    else:
    ...        print("Success: ", text)
    >>> test_random("It works!")
    """

    if func is None:
        return partial(
            retry,
            exception=exception,
            n_tries=n_tries,
            delay=delay,
            backoff=backoff,
            logger=logger,
        )

    @wraps(func)
    def wrapper(*args, **kwargs):
        ntries, ndelay = n_tries, delay

        while ntries > 1:
            try:
                return func(*args, **kwargs)
            except exception as e:
                msg = f"{str(e)}, Retrying in {ndelay} seconds..."
                if logger:
                    logging.warning(msg)
                else:
                    print(msg)
                time.sleep(ndelay)
                ntries -= 1
                ndelay *= backoff

        return func(*args, **kwargs)

    return wrapper

Answer 3

There are a couple of Python Packages:

1. backoff
2. tenacity

Example for backoff

import backoff

@backoff.on_exception(backoff.expo,
                      (MyPossibleException1,
                       MyPossibleException2))
def your_function(param1, param2):
    # Do something

Example for tenacity

from tenacity import wait_exponential, retry, stop_after_attempt

@retry(wait=wait_exponential(multiplier=2, min=2, max=30),  stop=stop_after_attempt(5))
def your_function(param1, param2):
   # Do something

Answer 4

There is another pip installable retry package: https://pypi.org/project/retry/

pip install retry

Answer 5

You can do something like the following:

def retry_call(function_name, args=None, kwargs=None, retries=3):
    pass_on_args = args if args else []
    pass_on_kwargs = kwargs if kwargs else {}
    for index in range(1, retries+1):
        try:
            return function_name(*pass_on_args, **pass_on_kwargs)
        except Exception as error:
            if index == retries:
                LOGGER.error("Failed %s attempts at calling: %s", retries, function_name)
                LOGGER.error("    args: %s", args)
                LOGGER.error("    kwargs: %s", kwargs)
                LOGGER.error("    error: %s", error)
                raise
            else:
                LOGGER.warning("Failed %d attempt(s) at calling: %s", index, function_name)
                LOGGER.warning("Retrying")

def get_variable
   with open(FILE) as info:
        for line in info:
            if 'Mode' in line:
            return mode

And then you would invoke the function:

mode = retry(get_variable)
print(mode)