Early Stop callback in keras

Question

How can one effectively stop the fit process of a training model via callback in keras? Thus far I have tried various approaches including the one below.

class EarlyStoppingCallback(tf.keras.callbacks.Callback):
    def __init__(self, threshold):
        super(EarlyStoppingCallback, self).__init__()
        self.threshold = threshold

    def on_epoch_end(self, epoch, logs=None):
        accuracy = logs["accuracy"]
        if accuracy >= self.threshold:
            print("Stopping early!")
            self.model.stop_training = True

The callback is executed, however the self.model.stop_training = True does not seem to have an effect. The print succeeds, but the model continues training. Any idea how to resolve this issue? My tensorflow version is: tensorflow==1.14.0

Could you share a [minimum reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of your code — Desmond Cheong, Jun 01 '20 at 16:49

Alexander Pivovarov · Accepted Answer · 2020-06-02T04:55:10.500

1

You're probably affected by the following issue: https://github.com/tensorflow/tensorflow/issues/37587.

In short - whenever model.predict or model.evaluate are called, model.stop_training is reset to False. I was able to reproduce this behavior using your EarlyStoppingCallback followed by another callback which was calling model.predict on some fixed dataset.

The workaround is to put callbacks which are calling model.predict or model.evaluate first before any callbacks which might want to set model.stop_training to True. It also looks like the issue was fixed in TF 2.2.

edited Jun 02 '20 at 04:55

answered Jun 02 '20 at 04:06

Alexander Pivovarov

4,850
1
11
34

Thank you :), this was very insightful. Problem could be immediately fixed. – Imago Jun 03 '20 at 21:42

Early Stop callback in keras

1 Answers1