Good accuracy despite high loss value

Question

During the training of a simple neural network binary classifier I get an high loss value, using cross-entropy. Despite this, accuracy's value on validation set holds quite good. Does it have some meaning? There is not a strict correlation between loss and accuracy?

I have on training and validation these values: 0.4011 - acc: 0.8224 - val_loss: 0.4577 - val_acc: 0.7826. This is my first attempt to implement a NN, and I just approached machine learning, so I'm not able to properly evaluate these results.

Answer 1

I have experienced a similar issue.

I have trained my neural network binary classifier with a cross entropy loss. Here the result of the cross entropy as a function of epoch. Red is for the training set and blue is for the test set.

By showing the accuracy, I had the surprise to get a better accuracy for epoch 1000 compared to epoch 50, even for the test set!

To understand relationships between cross entropy and accuracy, I have dug into a simpler model, the logistic regression (with one input and one output). In the following, I just illustrate this relationship in 3 special cases.

In general, the parameter where the cross entropy is minimum is not the parameter where the accuracy is maximum. However, we may expect some relationship between cross entropy and accuracy.

[ In the following, I assume that you know what is cross entropy, why we use it instead of accuracy to train model, etc. If not, please read this first: How do interpret an cross entropy score? ]

Illustration 1 This one is to show that the parameter where the cross entropy is minimum is not the parameter where the accuracy is maximum, and to understand why.

Here is my sample data. I have 5 points, and for example input -1 has lead to output 0.

Cross entropy. After minimizing the cross entropy, I obtain an accuracy of 0.6. The cut between 0 and 1 is done at x=0.52. For the 5 values, I obtain respectively a cross entropy of: 0.14, 0.30, 1.07, 0.97, 0.43.

Accuracy. After maximizing the accuracy on a grid, I obtain many different parameters leading to 0.8. This can be shown directly, by selecting the cut x=-0.1. Well, you can also select x=0.95 to cut the sets.

In the first case, the cross entropy is large. Indeed, the fourth point is far away from the cut, so has a large cross entropy. Namely, I obtain respectively a cross entropy of: 0.01, 0.31, 0.47, 5.01, 0.004.

In the second case, the cross entropy is large too. In that case, the third point is far away from the cut, so has a large cross entropy. I obtain respectively a cross entropy of: 5e-5, 2e-3, 4.81, 0.6, 0.6.

The $a$ minimizing the cross entropy is 1.27. For this $a$, we can show the evolution of cross entropy and accuracy when $b$ varies (on the same graph).

Illustration 2 Here I take $n=100$. I took the data as a sample under the logit model with a slope $a=0.3$ and an intercept $b=0.5$. I selected a seed to have a large effect, but many seeds lead to a related behavior.

Here, I plot only the most interesting graph. The $b$ minimizing the cross entropy is 0.42. For this $b$, we can show the evolution of cross entropy and accuracy when $a$ varies (on the same graph).

Here is an interesting thing: The plot looks like my initial problem. The cross entropy is rising, the selected $a$ becomes so large, however the accuracy continues to rise (and then stops to rise).

We couldn't select the model with this larger accuracy (first because here we know that the underlying model is with $a=0.3$!).

Illustration 3 Here I take $n=10000$, with $a=1$ and $b=0$. Now, we can observe a strong relationship between accuracy and cross entropy.

I think that if the model has enough capacity (enough to contain the true model), and if the data is large (i.e. sample size goes to infinity), then cross entropy may be minimum when accuracy is maximum, at least for the logistic model. I have no proof of this, if someone has a reference, please share.

Bibliography: The subject linking cross entropy and accuracy is interesting and complex, but I cannot find articles dealing with this... To study accuracy is interesting because despite being an improper scoring rule, everyone can understand its meaning.

Note: First, I would like to find an answer on this website, posts dealing with relationship between accuracy and cross entropy are numerous but with few answers, see: Comparable traing and test cross-entropies result in very different accuracies ; Validation loss going down, but validation accuracy worsening ; Doubt on categorical cross entropy loss function ; Interpreting log-loss as percentage ...

Answer 2

One important thing to note as well is that the cross entropy is not a bounded loss. Which means that a single very wrong prediction can potentially make your loss "blow up". In that sense it is possible that there are one or a few outliers that are classified extremely badly and that are making the loss explode, but at the same time your model is still learning on the rest of the dataset.

In the following example I use a very simple dataset in which there is an outlier in the test data. There are 2 classes "zero" and "one".

Here is how the dataset looks like:

As you can see the 2 classes are extremely easy to separate:above 0.5 it is class "zero". There is also a single outlier of class "one" in the middle of class "zero" only in the test set. This outlier is important as it will mess with the loss function.

I train a 1 hidden neural network on this dataset, you can see the results:

The loss starts increasing, but the accuracy continue to increase nonetheless.

Plotting a histogram of the loss function per samples shows clearly the issue: the loss is actually very low for most samples (the big bar at 0) and there is one outlier with a huge loss (small bar at 17). Since the total loss is the average you get a high loss on that set even though it is performing very well on all the points but one.

Bonus: Code for the data and model

import tensorflow.keras as keras
import numpy as np

np.random.seed(0)
x_train_2 = np.hstack([1/2+1/2*np.random.uniform(size=10), 1/2-1.5*np.random.uniform(size=10)])
y_train_2 = np.array([0,0,0,0,0,0,0,0,0,0, 1,1,1,1,1,1,1,1,1,1])
x_test_2 = np.hstack([1/2+1/2*np.random.uniform(size=10), 1/2-1.5*np.random.uniform(size=10)])
y_test_2 = np.array([0,0,0,1,0,0,0,0,0,0, 1,1,1,1,1,1,1,1,1,1])

keras.backend.clear_session()
m = keras.models.Sequential([
    keras.layers.Input((1,)),
    keras.layers.Dense(3, activation="relu"),
    keras.layers.Dense(1, activation="sigmoid")
])
m.compile(
    optimizer=keras.optimizers.Adam(lr=0.05), loss="binary_crossentropy", metrics=["accuracy"])
history = m.fit(x_train_2, y_train_2, validation_data=(x_test_2, y_test_2), batch_size=20, epochs=300, verbose=0)

TL;DR

Your loss might be hijacked by a few outliers, check the distribution of your loss function on individual samples of your validation set. If there are a cluster of values around the mean then you are overfitting. If there are just a few values very high above a low majority group then your loss is being affected by outliers :)

Answer 3

ahstat gives very good illustrations.

Inspired by these illustrations, i conclude to 2 possible reasons. 1. Model is too simple to extract required features for prediction. In your Illustration 1, it's a manifold problem and need to one more layer to get 100% accuracy. 2. Data has too many noisy label.(compare Illustration 1 and 3)

As for Illustration 2, it explains why we cannot add too much L1/L2 regularization on the model.

Answer 4

In categorical cross entropy case accuracy measures true positive i.e accuracy is discrete values, while the logloss of softmax loss so to speak is a continuous variable that measures the models performance against false negatives. A wrong prediction affects accuracy slightly but penalizes the loss disproportionately. Assuming you have a balanced dataset.

So what causes loss-vs-accuracy discrepancy ?

• when model predictions are bolder loss drops and accuracy stays constant. Implying model is performing well against its classes

• a single wrong prediction with confidence will drop accuracy slightly but loss will increase i.e over fitted model may have good accuracy but poor loss

For categorical cross entropy loss a dumb model that just guesses should have a loss of y=-ln(1/n) where n is number or classes that are balanced. Further apply probability with imbalance between your classes to calculate expected chance logloss first as baseline. Once you know that you can then evaluate how well the model is trained and use loss as a proxy to accuracy to infer model performance.