How can I limit the maximum running time for a unit test?

Question

I am currently running some unit tests that might either take a long time before failing or run indefinitely. In a successful test run they will always complete within a certain amount of time.

Is it possible to create a pytest unit test that will fail if it does not complete within a certain amount of time?

score 93 · Accepted Answer · edited Feb 07 '17 at 10:44

93

you can install the pytest-timeout plugin and then mark your test functions with a timeout in seconds.

@pytest.mark.timeout(300)
def test_foo():
   pass

Look at the plugin download and usage instructions at https://pypi.python.org/pypi/pytest-timeout

edited Feb 07 '17 at 10:44

shuttle87

15,466
11
77
106

answered Oct 22 '13 at 20:17

Mrinal Shukla

1,010
8
7

is this doable without installing an extra package? – Gulzar Jan 24 '21 at 16:09
@Gulzar if you really need to you could vendor in this dependency or write a similar decorator yourself. – shuttle87 Jan 28 '21 at 05:40
This doesn't seem to work in my case: but admittedly I am using pytest-qt and the cause of the "hang" is a `QMessageBox` calling `question`, which causes the test to just stop mid-flow if there is no user input. Maybe pytest-qt has something specific... ? – mike rodent May 05 '22 at 17:31
[pytest-timeout](https://github.com/pytest-dev/pytest-timeout): 5 PRs pending, 53 forks, and currently broken build. I'd think hard before using it. – Abhijit Sarkar Jul 06 '23 at 04:28

score 1 · Answer 2 · answered Oct 11 '21 at 13:26

1

Bash functionality can be used:

EXIT_CODE=0
TIME_LIMIT=60

timeout $TIME_LIMIT pytest ... || EXIT_CODE=$?

if [ $EXIT_CODE -ne 0 ]; then
    
    echo "Your error message to log"
    
    ...
    
    exit $EXIT_CODE
    
fi

answered Oct 11 '21 at 13:26

Konstantin

19
1

score 0 · Answer 3 · answered Feb 18 '23 at 02:36

Here is a lightweight way that you can use in a doctest or directly in your script and without any extra dependency, inspired from a question on how to limit execution time for a function

import signal
from contextlib import contextmanager


@contextmanager
def time_limit(seconds):
    def signal_handler(signum, frame):
        raise TimeoutError
    signal.signal(signal.SIGALRM, signal_handler)
    signal.alarm(seconds)
    try:
        yield
    finally:
        signal.alarm(0)

def efficiency_test(function, *args, **kwargs):
    try:
        with time_limit(2):
            return (True, function(*args, **kwargs))
    except TimeoutError as e:
        return (False, None)

With example usage:

import signal
from contextlib import contextmanager


@contextmanager
def time_limit(seconds):
    def signal_handler(signum, frame):
        raise TimeoutError
    signal.signal(signal.SIGALRM, signal_handler)
    signal.alarm(seconds)
    try:
        yield
    finally:
        signal.alarm(0)

def simple(a, b):
    return a + b

def complex_(a, b):
    for i in range(100000000):
        a += b
    return a

def efficiency_test(function, *args, **kwargs):
    try:
        with time_limit(2):
            return (True, function(*args, **kwargs))
    except TimeoutError as e:
        return (False, None)

if __name__ == "__main__":
    print(efficiency_test(simple, 1, 2))
    print(efficiency_test(complex_, 1, 2))

With output:

(True, 3)
(False, None)

How can I limit the maximum running time for a unit test?

3 Answers3

Linked