In Python why is `KeyboardInterrupt` preventing the pickling of this object?

Question

I need a better method or I need to learn how to use the method I'm using more correctly. Normally, when I press cntrl-c my work gets pickled, but one time it did not. The subsequent time that I tried to open the pickle I got a ran out of input error. I need to know why this happened so that it does not happen again. As I'm running my code, every one hundred loops I will save the pickle and if I hit KeyboardInterrupt, theoretically my work is supposed to be pickled before the program stops. My hunch is that if I press KeyboardInterrupt while pickle.dump(obj, temp) is being executed then the file will overwrite the old file but as it's overwriting if the program is killed in the middle of the overwrite then the file will be sort of half-written. What I also do not understand is that after I hit KeyboardInterrupt the program should execute the line print("help me") but it's not, at least on the sole occasion where I tried that.

import pickle

def save_pickle(obj, file):
    temp = open(file, "wb")
    ##i have a feeling that if i hit keyboard interrupt
    ##while the below line is being executed that it won't
    ## save the pickle properly
    pickle.dump(obj, temp)
    temp.close()

class help_me():
    def __init__(self):
        pass

    def func1(self):
        try:
            self.func2()
        except KeyboardInterrupt:
            pass
        print('help me')  # this line did not get activated
        save_pickle(obj, file)

    def func2(self):
        #this function will take several days
        for i in range(600_000):
            time.sleep(1)
            if i % 100 == 0:
                save_pickle(obj, file)

I mean, yes, if your program is terminated halfway through saving, then you don't save a complete object. And your `save_pickle` call is not within a `KeyboardInterrupt` catch block. It sounds like you already know what the problem is; what do you need from us? — Silvio Mayolo, Aug 07 '23 at 19:19
What do you want to happen when you program receives SIGINT midway through writing the pickle file? Finish the current object or discard it? — Brian61354270, Aug 07 '23 at 19:22
@SilvioMayolo `save_pickle()` is called from `func2()`, and `func2()` is called within `try/except`. That should catch the interrupt and then it should continue at `print('help me')` — Barmar, Aug 07 '23 at 19:22
@Silvio, it sounds like I know what the problem is, but sounds can be deceiving. — bobsmith76, Aug 07 '23 at 19:23
@SilvioMayolo, I though `save_pickle` would be within the `try/except` because if `func2` is within `try/except` and if `func2` calls `save_pickle` then `save_pickle` is within `func2`. After all, if the program is not within the `try/except` then why would it stop on `cntrl-c`? — bobsmith76, Aug 07 '23 at 19:27
@bobsmith76 You probably want something like this: before the loop, register a temporary [signal handler](https://stackoverflow.com/q/1112343/11082165) for SIGINT that sets a "stop" flag instead of raising KeyboardInterrupt. In the loop, check for the stop flag, and break when it's set. After the loop, [reset the signal handler](https://stackoverflow.com/q/22916783/11082165) — Brian61354270, Aug 07 '23 at 19:28
@Brian61354270, you're assuming a level of expertise that I don't have. Could you write up an answer and explain your thoughts in more detail. — bobsmith76, Aug 07 '23 at 19:29
I don't see how you do anything, at all. Once you call `func2` your program is locked up for 600,000 seconds, or 10,000 minutes! — OneMadGypsy, Aug 07 '23 at 19:42
@Brian61354270 - I didn't miss the "worst possible way to do this" part. — OneMadGypsy, Aug 07 '23 at 19:45

score 1 · Accepted Answer · answered Aug 07 '23 at 19:43

As noted in the comments, if you terminate your program with SIGINT (i.e. CTRL+C) mid-way through pickling an object to a file, you may end up with a partially written (corrupt) file.

To avoid this, you can customize how your program responds to SIGINT so that the critical parts don't stop abruptly.

Before the loop, register a temporary signal handler for SIGINT that sets a "stop" flag instead of raising KeyboardInterrupt. In the loop, check for the stop flag, and break when it's set. After the loop, you can reset the signal handler to whatever it was before the loop.

def func2(self):
    stop = False
  
    def handle_sigint(_sig, _frame)
        nonlocal stop
        print("setting stop flag")
        stop = True

    original_handler = signal.getsignal(signal.SIGINT)
    signal.signal(signal.SIGINT, handle_sigint)

    while not stop:
        time.sleep(1)
        save_pickle(obj, file)

    signal.signal(signal.SIGINT, original_handler)

score 1 · Answer 2 · answered Aug 07 '23 at 19:53

First, I have a few comments on your code:

In the function save_pickle, if any exception happens while executing pickle.dump(obj, temp) such as a KeyboardInterrupt, the temp.close() statement won't be executed. To prevent that, you can use a context manager.
Declaring the help_me class does not follow Python's idioms. Lose the parentheses and use pascal case notation like HelpMe.
Instead of except: pass in func1 you should better use contextlib.suppress(KeyboardInterrupt): ....
The line with print('help me') statement should get executed unless you press ctrl-c more than once. If you want that to happen anyway, you can use a finally statement.
You can also lose the __init__ function since it doesn't do anything.

To wrap up, here's my correction of the code, with just a few modifications:

import pickle
import time

def save_pickle(obj, file_address):
    with open(file_address, "wb") as file_:
        pickle.dump(obj, file_)

class HelpMe:
    def func1(self):
        key_board_interrupt_happened = False
        try:
            self.func2()
        except KeyboardInterrupt:
            key_board_interrupt_happened = True
        finally:
            print('help me')
            if key_board_interrupt_happened:
                pass
                # You can do the final save of what is left here.

    def func2(self):
        for i in range(600_000):
            time.sleep(1)
            if i % 100 == 0:
                save_pickle(object(), "file_address")

If you want the final save to happen after any kind of exception, you can loose the ``except`` statement. A simple ``trt-finally`` will do. — Sadegh Moayedizadeh, Aug 07 '23 at 19:56
About pascal_case notation, I never use capital letters in any code I write, that way I don't have to remember whether a word has capital letters or not. This reduces complexity and makes things easier to remember. The sooner people learn this simple fact, the better. — bobsmith76, Aug 08 '23 at 04:11

In Python why is `KeyboardInterrupt` preventing the pickling of this object?

2 Answers2