Cython, Python and KeyboardInterrupt ignored

Question

Is there a way to interrupt (Ctrl+C) a Python script based on a loop that is embedded in a Cython extension?

I have the following python script:

def main():

    # Intantiate simulator
    sim = PySimulator()
    sim.Run()

if __name__ == "__main__":
    # Try to deal with Ctrl+C to abort the running simulation in terminal
    # (Doesn't work...)
    try:
        sys.exit(main())
    except (KeyboardInterrupt, SystemExit):
        print '\n! Received keyboard interrupt, quitting threads.\n'

This runs a loop that is part of a C++ Cython extension. Then, while pressing Ctrl+C, the KeyboardInterrupt is thrown but ignored, and the program keeps going until the end of the simulation.

The work around I found, is to handle the exception from within the extension by catching the SIGINT signal :

#include <execinfo.h>
#include <signal.h>

static void handler(int sig)
{
  // Catch exceptions
  switch(sig)
  {
    case SIGABRT:
      fputs("Caught SIGABRT: usually caused by an abort() or assert()\n", stderr);
      break;
    case SIGFPE:
      fputs("Caught SIGFPE: arithmetic exception, such as divide by zero\n",
            stderr);
      break;
    case SIGILL:
      fputs("Caught SIGILL: illegal instruction\n", stderr);
      break;
    case SIGINT:
      fputs("Caught SIGINT: interactive attention signal, probably a ctrl+c\n",
            stderr);
      break;
    case SIGSEGV:
      fputs("Caught SIGSEGV: segfault\n", stderr);
      break;
    case SIGTERM:
    default:
      fputs("Caught SIGTERM: a termination request was sent to the program\n",
            stderr);
      break;
  }
  exit(sig);

}

Then :

signal(SIGABRT, handler);
signal(SIGFPE,  handler);
signal(SIGILL,  handler);
signal(SIGINT,  handler);
signal(SIGSEGV, handler);
signal(SIGTERM, handler);

Can't I make this work from Python, or at least from Cython instead ? As I am about to port my extension under Windows/MinGW, I would appreciate to have something less Linux specific.

Answer 1

You have to periodically check for pending signals, for example, on every Nth iteration of the simulation loop:

from cpython.exc cimport PyErr_CheckSignals

cdef Run(self):
    while True:
        # do some work
        PyErr_CheckSignals()

PyErr_CheckSignals will run signal handlers installed with signal module (this includes raising KeyboardInterrupt if necessary).

PyErr_CheckSignals is pretty fast, it's OK to call it often. Note that it should be called from the main thread, because Python runs signal handlers in the main thread. Calling it from worker threads has no effect.

Explanation

Since signals are delivered asynchronously at unpredictable times, it is problematic to run any meaningful code directly from the signal handler. Therefore, Python queues incoming signals. The queue is processed later as part of the interpreter loop.

If your code is fully compiled, interpreter loop is never executed and Python has no chance to check and run queued signal handlers.

Answer 2

If you are trying to handle KeyboardInterrupt in code that releases the GIL (for example, because it uses cython.parallel.prange), you will need to re-acquire the GIL to call PyErr_CheckSignals. The following snippet (adapted from @nikita-nemkin's answer above) illustrates what you need to do:

from cpython.exc cimport PyErr_CheckSignals
from cython.parallel import prange

cdef Run(self) nogil:
    with nogil:
        for i in prange(1000000)
            # do some work but check for signals every once in a while
            if i % 10000 == 0:
                with gil:
                    PyErr_CheckSignals()

Answer 3

Release GIL when Cython runs parts that do not interface with Python, run loop in the main thread (sleep or check the simulation status), and call sim.Stop() (that can set some flag that your simulation can check periodically) in the except close.

Answer 4

Yes, using the macros sig_on and sig_off from the package cysignals:

from cysignals.signals cimport sig_on, sig_off

def foo():
    sig_on()
    call_c_code_that_takes_long()
    sig_off()

The macros sig_on and sig_off are declared as functions in cysignals/signals.pxd, and defined as macros in cysignals/macros.h in terms of the macro _sig_on_ (defined in terms of the functions _sig_on_prejmp and _sig_on_postjmp) and the function _sig_off_. The signal handler for keyboard interrupts (SIGINT) is installed here, and the implementation rationale is outlined here.

As of cysignals == 1.6.5, only POSIX systems are supported. Cython's conditional compilation can be used in order to follow this approach wherever cysignals is available, and allow compiling on non-POSIX systems too (without Ctrl-C working on those systems).

In the script setup.py:

compile_time_env = dict(HAVE_CYSIGNALS=False)
# detect `cysignals`
if cysignals is not None:
    compile_time_env['HAVE_CYSIGNALS'] = True
...
c = cythonize(...,
              compile_time_env=compile_time_env)

and in the relevant *.pyx file:

IF HAVE_CYSIGNALS:
    from cysignals.signals cimport sig_on, sig_off
ELSE:
    # for non-POSIX systems
    noop = lambda: None
    sig_on = noop
    sig_off = noop

See also this answer.