Calculating micro F-1 score in keras

Question

I have a dataset with 15 imbalanced classes and trying to do multilabel classification with keras.

I am trying to use micro F-1 score as a metric.

My model:

# Create a VGG instance
model_vgg = tf.keras.applications.VGG19(weights = 'imagenet', pooling = 'max', include_top = False, 
input_shape = (512, 512, 3))

# Freeze the layers which you don't want to train. 
for layer in model_vgg.layers[:-5]:
layer.trainable = False

# Adding custom Layers 
x = model_vgg.output
x = Flatten()(x)
x = Dense(1024, activation = "relu")(x)
x = Dropout(0.5)(x)
x = Dense(1024, activation = "relu")(x)
predictions = Dense(15, activation = "sigmoid")(x)

# creating the final model 
model_vgg_final = Model(model_vgg.input, predictions)

# Print the summary
model_vgg_final.summary()

For F1 score I use the custom metric from this question

from keras import backend as K

def f1(y_true, y_pred):
    def recall(y_true, y_pred):
        """Recall metric.

    Only computes a batch-wise average of recall.

    Computes the recall, a metric for multi-label classification of
    how many relevant items are selected.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

    def precision(y_true, y_pred):
        """Precision metric.

    Only computes a batch-wise average of precision.

    Computes the precision, a metric for multi-label classification of
    how many selected items are relevant.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

    precision = precision(y_true, y_pred)
    recall = recall(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))

I use binary cross-entropy and a custom F-1, while compiling a model

# Compile a model
model_vgg_final.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = [f1]) 

I monitor F-1 for early stopping

# Early stopping
early_stopping = EarlyStopping(monitor = 'f1', patience = 5)

# Training the model
history_vgg = model_vgg_final.fit(train_generator, steps_per_epoch = 10, epochs = 30, verbose = 1, 
callbacks = [early_stopping], validation_data = valid_generator)

How to update this custom function and get micro F-1 as a metric? Also appreciate tips about my approach.

There is information in scikit-learn documentation, but not sure how to incorporate it in keras

Answer 1

Good question.

The link that you provided there points to how the metrics were calculated in an older version of Keras (bear with me, short explanation). The problem was, in older Keras (1.X) that the metrics were calculated batch-wise, which of course would lead to incorrect global results. In Keras 2.X, the built-it metrics were removed.

However, there are solutions to your problem.

1. You can implement you own, custom callback. You can check my answer here, it is guaranteed to work in TensorFlow 2.x: How to get other metrics in Tensorflow 2.0 (not only accuracy)?
2. You can use tensorflow-addons --> pip install tensorflow-addons. TensorFlow addons is a very good package which incorporates multiple functionalities and features, which are unavailable in the base TensorFlow package. Here, the F1Score is a built-in metric, so you can use it directly.

Example:

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.00001),
              loss=tf.keras.losses.BinaryCrossentropy(),
              metrics=[tf.keras.metrics.BinaryAccuracy(),                                
                      tfa.metrics.F1Score(num_classes=number_of_classes, 
                      average='micro',
                      threshold=0.5)])

Please note the usage of 'micro' parameter which in fact represents exactly what you want, the micro f1-score.