Allowing Ctrl-C to interrupt a python C-extension

Question

I'm running some computationally heavy simulation in (home-made) C-based python extensions. Occasionally I get stuff wrong and would like to terminate a simulation. However, Ctrl-C doesn't seem to have any effect (other than printing ^C to the screen) so I have to kill the process using kill or the system monitor.

As far as I can see python just waits for the C extension to finish and doesn't really communicate with it during this time.

Is there a way to make this work?

Update: The main answers (for my specific problem) turned out to be: 1. rewrite the code to regularly pass control back to the caller (answer Allowing Ctrl-C to interrupt a python C-extension below), or 2. Use PyErr_CheckSignals() (answer https://stackoverflow.com/a/33652496/423420 below)

Answer 1

However, Ctrl-C doesn't seem to have any effect

Ctrl-C in the shell sends SIGINT to the foreground process group. python on receiving the signal sets a flag in C code. If your C extension runs in the main thread then no Python signal handler will be run (and therefore you won't see KeyboardInterrupt exception on Ctrl-C) unless you call PyErr_CheckSignals() that checks the flag (it means: it shouldn't slow you down) and runs Python signal handlers if necessary or if your simulation allows Python code to execute (e.g., if the simulation uses Python callbacks). Here's a code example of an extension module for CPython created using pybind11 suggested by @Matt:

PYBIND11_MODULE(example, m)
{
    m.def("long running_func", []()
    {
        for (;;) {
            if (PyErr_CheckSignals() != 0)
                throw py::error_already_set();
            // Long running iteration
        }
    });
}

If the extension runs in a background thread then it is enough to release GIL (to allow Python code to run in the main thread that enables the signal handlers to run). PyErr_CheckSignals() always returns 0 in a background thread.

Related: Cython, Python and KeybordInterrupt ingored

Answer 2

Python has a signal handler installed on SIGINT which simply sets a flag that is checked by the main interpreter loop. For this handler to work properly, the Python interpreter has to be running Python code.

You have a couple of options available to you:

1. Use Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release the GIL around your C extension code. You cannot use any Python functions when not holding the GIL, but Python code (and other C code) may run concurrently with your C thread (true multithreading). A separate Python thread can execute alongside the C extension and catch Ctrl+C signals.
2. Set up your own SIGINT handler and call the original (Python) signal handler. Your SIGINT handler can then do whatever it needs to do to cancel the C extension code and return control to the Python interpreter.

Answer 3

There is an alternative way to solve this problem if you do not want your C Extension (or ctypes DLL) to be tied to Python, such as a case where you want to create a C library with bindings in multiple languages, you must allow your C Extension to run for long periods, and you can modify the C Extension:

Include the signal header in the C Extension.

#include <signal.h>

Create a signal handler typedef in the C Extension.

typedef void (*sighandler_t)(int);

Add signal handlers in the C extension that will perform the actions necessary to interrupt any long running code (set a stop flag, etc.), and save the existing Python signal handlers.

sighandler_t old_sig_int_handler = signal(SIGINT, your_sig_handler);
sighandler_t old_sig_term_handler = signal(SIGTERM, your_sig_handler);

Restore the existing signal handlers whenever the C extension returns. This step ensures that the Python signal handlers are re-applied.

signal(SIGINT, old_sig_int_handler);
signal(SIGTERM, old_sig_term_handler);

If the long-running code is interrupted (flag, etc.), return control to Python with a return code indicating the signal number.

return SIGINT;

In Python, send the signal received in the C extension.

import os
import signal

status = c_extension.run()

if status in [signal.SIGINT, signal.SIGTERM]:
    os.kill(os.getpid(), status)

Python will perform the action you are expecting, such as raising a KeyboardInterrupt for SIGINT.

Answer 4

Not elegant, but the only approach I found that also interrupts external library calls in C++ and kills any running child processes.

#include <csignal>
#include <pybind11/pybind11.h>

void catch_signals() {
  auto handler = [](int code) { throw std::runtime_error("SIGNAL " + std::to_string(code)); };
  signal(SIGINT, handler);
  signal(SIGTERM, handler);
  signal(SIGKILL, handler);
}

PYBIND11_MODULE(example, m)
{
    m.def("some_func", []()
    {
        catch_signals();
        // ...
    });
}

import sys

from example import some_func

try:
    some_func()
except RuntimeError as e:
    if "SIGNAL" in str(e):
        code = int(str(e).rsplit(" ", 1)[1])
        sys.exit(128 + code)
    raise

Answer 5

I would redesign the C extensions so that they don't run for a long period.

So, split them into more elementary steps (each running for a short period of time, e.g. 10 to 50 milliseconds), and have these more elementary steps called by Python code.

continuation passing style might be relevant to understand, as a programming style...