Calculate the model compiled loss and metrics from predicted scores

Question

After training the model, I use the predict method to infer the scores from my testing data.

From the predicted scores how can I use the model compiled loss and metrics to calculate the loss of my predictions?

What I have tried

Based on Customizing what happens in fit() guide I tried using the compiled_loss method

y_pred = model.predict(x_test)
model.compiled_loss(y_test, y_pred, regularization_losses=model.losses)

But it returns the error

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-6-3eb62dca0b87> in <module>()
      1 y_pred = model.predict(x_test)
----> 2 loss = model.compiled_loss(y_test, y_pred)

1 frames

/usr/local/lib/python3.7/dist-packages/keras/engine/compile_utils.py in match_dtype_and_rank(y_t, y_p, sw)
    673 def match_dtype_and_rank(y_t, y_p, sw):
    674   """Match dtype and rank of predictions."""
--> 675   if y_t.shape.rank == 1 and y_p.shape.rank == 2:
    676     y_t = tf.expand_dims(y_t, axis=-1)
    677   if sw is not None:

AttributeError: 'tuple' object has no attribute 'rank'

How to reproduce

I used the Simple MNIST convnet example followed by

y_pred = model.predict(x_test)
model.compiled_loss(y_test, y_pred, regularization_losses=model.losses)

to reproduce the error

About my problem

I am validating my data on a custom metric. However some Keras users recommended that global metrics should not be averaged by batch, instead, calculated from the predicted scores for the whole validation data in a Callback.

See:

Issue #5794

How to calculate F1 Macro in Keras?

The bad solution to this is to calculate the loss and metrics from the evaluate method, and my custom metric from predict. The problem with this is that I am running the inference twice.

A less worse solution is to implement my loss function separately so it can work from the predicted scores.

See:

Calculate loss in Keras without running the model

The issue with this is that it gives me less flexibility to choose loss functions because I have to implement every loss function separately in the Callback later.

But I really wonder, isn't the compiled loss and metrics accessible somewhere already?

Answer 1

Here's a function that can calculate the metrics given the previously predicted output:

import tensorflow as tf

def calculate_metrics(model, y, y_pred, sample_weight=None):
    y = tf.convert_to_tensor(y)
    y_pred = tf.convert_to_tensor(y_pred)
    if not sample_weight is None:
        sample_weight = tf.convert_to_tensor(sample_weight)
    model.compute_loss(x=None, y=y, y_pred=y_pred, sample_weight=sample_weight)
    metrics = model.compute_metrics(x=None, y=y, y_pred=y_pred,
                                    sample_weight=sample_weight)
    for k,v in metrics.items():
        metrics[k] = float(v)
    return metrics

It expects y, y_pred, and sample_weight (if provided) to be numpy arrays. This means that they need to fit in memory at once. It also means that the metric values could be slightly different than what is returned by model.evaluate, but I think you understand that, given your comment that "global metrics should not be averaged by batch."

---

But it returns the error

AttributeError: 'tuple' object has no attribute 'rank'

model.compiled_loss (as well as model.compute_loss and model.compute_metrics) expects the inputs to each be a Tensor. Tensors have an attribute shape, which is of type TensorShape, which in turn has an attribute rank. I expect that your y_test was an ndarray, which also has an attribute shape, but that is of type tuple, which does not have an attribute rank. This is why my function converts the inputs to Tensors. The metric values returned are also Tensors, so I manually convert each of those to an ordinary python float.

---

But I really wonder, isn't the compiled loss and metrics accessible somewhere already?

The metrics are available in the History object returned by the model.fit method, so if you're evaluating a newly trained model, you don't need to recalculate the metrics. However, if you're loading a previously trained model and want to get the metrics, you need to either use model.evaluate or calculate the metrics from the result of model.predict as we do here. Taking a look at the code, you can compare model.test_step and model.predict_step, respectively. Note that model.predict_step just runs the inference and returns the prediction, so the metrics are never calculated when you run model.predict. A better solution than the one I provide would be to combine elements of model.predict into model.evaluate so that it can optionally return the y_pred instead of discarding it after the metrics are calculated, but that would require modifying tensorflow code.