Timeout function using threading in python does not work

Question

I have found a code creating a timeout function here, which does not seem to work. The complete test code is below:

def timeout(func, args=(), kwargs={}, timeout_duration=1, default=None):
    import threading
    class InterruptableThread(threading.Thread):
        def __init__(self):
            threading.Thread.__init__(self)
            self.result = None

        def run(self):
            try:
                self.result = func(*args, **kwargs)
            except:
                self.result = default

    it = InterruptableThread()
    it.start()
    it.join(timeout_duration)
    if it.isAlive():
        return default
    else:
        return it.result


def foo():
    while True:
        pass

timeout(foo,timeout_duration=3)

Expected behavior: code ends within 3 seconds. Where is the problem?

score 13 · Accepted Answer · edited May 23 '17 at 11:53

A thread can not gracefully kill another thread, so with your current code, foo never terminates. (With thread.daemon = True the Python program will exit when only daemon threads are left, but that does not allow you to terminate foo without also terminating the main thread.)

Some people have tried to use signals to halt execution, but this may be unsafe in some cases.

If you can modify foo, there are many solutions possible. For instance, you could check for a threading.Event to break out of the while-loop.

But if you can not modify foo, you could run it in a subprocess using the multiprocessing module since unlike threads, subprocesses can be terminated. Here is an example of how that might look:

import time
import multiprocessing as mp

def foo(x = 1):
    cnt = 1
    while True:
        time.sleep(1)
        print(x, cnt)
        cnt += 1

def timeout(func, args = (), kwds = {}, timeout = 1, default = None):
    pool = mp.Pool(processes = 1)
    result = pool.apply_async(func, args = args, kwds = kwds)
    try:
        val = result.get(timeout = timeout)
    except mp.TimeoutError:
        pool.terminate()
        return default
    else:
        pool.close()
        pool.join()
        return val


if __name__ == '__main__':
    print(timeout(foo, kwds = {'x': 'Hi'}, timeout = 3, default = 'Bye'))
    print(timeout(foo, args = (2,), timeout = 2, default = 'Sayonara'))

yields

('Hi', 1)
('Hi', 2)
('Hi', 3)
Bye
(2, 1)
(2, 2)
Sayonara

Note that this has some limitations too.

subprocesses receive a copy of the parent processes' variables. If you modify a variable in a subprocess, it will NOT affect the parent process. If your function func needs to modify variables, you will need to use a shared variable.
arguments (passed through args) and keywords (kwds) must be picklable.
processes are more resource-heavy than threads. Usually, you only want to create a multiprocessing Pool once at the beginning of a program. This timeout function creates a Pool every time you call it. This was necessary since we needed pool.terminate() to terminate foo. There might be a better way, but I haven't thought of it.

Thanks, it works well for me. However, because of the returns it seems `pool.close()` and `pool.join()` are never called. — Jabba, Sep 26 '13 at 17:16

score 3 · Answer 2 · edited May 23 '17 at 12:00

3

You need to turn it into a daemon thread:

it = ...
it.daemon = True
it.start()

Otherwise it's created as a user thread, and the process won't get stopped until all the user threads have finished.

Note that with your implementation the thread will continue to run and consume resources even after you've timed out waiting for it. CPython's Global Interpreter Lock could exacerbate the issue further.

edited May 23 '17 at 12:00

Community

1
1

answered Dec 11 '12 at 13:13

NPE

486,780
108
951
1,012

Oh so maybe this code it not what I am looking for. I am looking for a function `timeout` that uses a different function `foo` (API as in the code above), that returns the result of `foo` as soon `foo` has ended, or a default value otherwise (as in the example). In both cases, I do not want to have something run, Maybe the use of `signal` is better? I have no idea what the Global Interpreter Lock is, but it is way too complicated. – Alex Dec 11 '12 at 13:22

score 2 · Answer 3 · answered Jul 06 '20 at 09:14

The benefit of using multiprocessing is the processes do not share memory and whatever happens in the child remains limited to that function and will not cause other process to terminate. The easiest way to add a timeout of 3s to a child process is:

import multiprocessing

def my_child():
    function here

process = multiprocessing.Process(target=my_child)
process.daemon = True
process.start()

process.join(3)
if process.is_alive():
    process.terminate()

Timeout function using threading in python does not work

3 Answers3

Linked

Related