I am training a simple neural network on the CIFAR10 dataset. After some time, validation loss started to increase, whereas validation accuracy is also increasing. The test loss and test accuracy continue to improve.

How is this possible? It seems that if validation loss increase, accuracy should decrease.

P.S. There are several similar questions, but nobody explained what was happening there.enter image description here

  • 2
    $\begingroup$ You can check some hints to understand in my answer here: stats.stackexchange.com/questions/258166/… $\endgroup$
    – ahstat
    Commented May 28, 2017 at 14:20
  • 1
    $\begingroup$ @ahstat I understand how it's technically possible, but I don't understand how it happens here. $\endgroup$ Commented May 28, 2017 at 15:16
  • $\begingroup$ The 'illustration 2' is what I and you experienced, which is a kind of overfitting. For my particular problem, it was alleviated after shuffling the set. $\endgroup$
    – ahstat
    Commented May 28, 2017 at 15:28
  • 1
    $\begingroup$ @ahstat There're a lot of ways to fight overfitting. For example, I might use dropout. What I am interesting the most, what's the explanation for this. I.e. why is it increasing so gradually and only up. $\endgroup$ Commented May 28, 2017 at 17:14

6 Answers 6


Other answers explain well how accuracy and loss are not necessarily exactly (inversely) correlated, as loss measures a difference between raw output (float) and a class (0 or 1 in the case of binary classification), while accuracy measures the difference between thresholded output (0 or 1) and class. So if raw outputs change, loss changes but accuracy is more "resilient" as outputs need to go over/under a threshold to actually change accuracy.

However, accuracy and loss intuitively seem to be somewhat (inversely) correlated, as better predictions should lead to lower loss and higher accuracy, and the case of higher loss and higher accuracy shown by OP is surprising. I have myself encountered this case several times, and I present here my conclusions based on the analysis I had conducted at the time. I stress that this answer is therefore purely based on experimental data I encountered, and there may be other reasons for OP's case.

Let's consider the case of binary classification, where the task is to predict whether an image is a cat or a dog, and the output of the network is a sigmoid (outputting a float between 0 and 1), where we train the network to output 1 if the image is one of a cat and 0 otherwise. In this case, two phenomenons can happen at the same time :

enter image description here

  1. Some images with borderline predictions get predicted better and so their output class changes to the correct one (image C in the figure). This is the classic "loss decreases while accuracy increases" behavior that we expect when training is going well.

  2. Some images with very bad predictions keep getting worse (image D). This leads to a less classic "loss increases while accuracy stays the same".
    You might think that this effect should be "countered" by the inverse phenomenon of images with very good predictions getting better (images A and B, note that their output has increased the same amount than the one of image D has decreased (0.02)). However, when using cross-entropy loss for classification (as it is usually done), bad predictions are penalized much more strongly than good predictions are rewarded. For a cat image (ground truth : 1), the loss is $-log(output)$, so slightly better predictions for images already close to 1 don't change the mean loss much (as the difference between log(<number close to 1>) and log(<number slightly closer to 1>) is low), while a single misclassified cat image getting worse can "blow up" your mean loss (as the difference between log(<number close to 0>) and log(<number slightly closer to 0>) is high). See this answer for further illustration of this phenomenon.
    (Getting increasing loss and stable accuracy could also be caused by good predictions being classified a little worse, but I find it less likely because of this loss "asymmetry").

So I think that when both accuracy and loss are increasing, the network is starting to overfit, and both phenomena are happening at the same time. The network is starting to learn patterns only relevant for the training set and not great for generalization, leading to phenomenon 2 where some images from the validation set get predicted really wrong (image D), with an effect amplified by the "loss asymmetry". However, it is at the same time still learning some patterns which are useful for generalization (phenomenon one, "good learning") as more and more images are being correctly classified (images A, B, C).

I sadly have no answer for whether or not this "overfitting" is a bad thing in this case: should we stop the learning once the network is starting to learn spurious patterns, even though it's continuing to learn useful ones along the way?

Finally, I think this effect can be further obscured in the case of multi-class classification, where the network at a given epoch might be severely overfit on some classes but still learning on others.

  • 1
    $\begingroup$ I got a very odd pattern where both loss and accuracy decreases. It doesn't seem to be overfitting because even the training accuracy is decreasing. Any ideas what might be happening? $\endgroup$ Commented Aug 13, 2020 at 15:03
  • $\begingroup$ Thank you for the explanations @Soltius. Out of curiosity - do you have a recommendation on how to choose the point at which model training should stop for a model facing such an issue? $\endgroup$ Commented Mar 16, 2022 at 6:49

Accuracy evaluated by comparing the highest softmax output and the correctly labeled class. It is not dependent on how high the softmax output is. To make it clearer, here are some numbers:

Suppose there are 2 classes - horse and dog. For our case, the correct class is horse . Now, the output of the softmax is [0.9, 0.1]. For this, the loss loss ~0.37. The classifier will predict that it is a horse. Take another case where the softmax output is [0.6, 0.4]. Loss ~0.6. The classifier will still predict that it is a horse. But surely, the loss has increased. So, it is all about the output distribution.

  • 6
    $\begingroup$ Observation: in your example, the accuracy doesnt change. It's still 100%. Do you have an example where loss decreases, and accuracy decreases too? $\endgroup$ Commented Dec 30, 2017 at 17:04
  • 2
    $\begingroup$ I would like to understand this example a bit more. First things first, there are three classes and the softmax has only 2 outputs. Why so? Should it not have 3 elements? $\endgroup$
    – JohnJ
    Commented Jun 11, 2020 at 12:47
  • $\begingroup$ @JohnJ I corrected the example and submitted an edit so that it makes sense. Thanks for pointing this out, I was starting to doubt myself as well. $\endgroup$
    – Dorian
    Commented Aug 18, 2021 at 8:50

Many answers focus on the mathematical calculation explaining how is this possible. But they don't explain why it becomes so. And they cannot suggest how to digger further to be more clear.

I have 3 hypothesis. And suggest some experiments to verify them. Hopefully it can help explain this problem.

  1. Label is noisy. Compare the false predictions when val_loss is minimum and val_acc is maximum. Check whether these sample are correctly labelled.
  2. [Less likely] The model doesn't have enough aspect of information to be certain. Experiment with more and larger hidden layers.
  3. [A very wild guess] This is a case where the model is less certain about certain things as being trained longer. Such situation happens to human as well. When someone started to learn a technique, he is told exactly what is good or bad, what is certain things for (high certainty). When he goes through more cases and examples, he realizes sometimes certain border can be blur (less certain, higher loss), even though he can make better decisions (more accuracy). And he may eventually gets more certain when he becomes a master after going through a huge list of samples and lots of trial and errors (more training data). So in this case, I suggest experiment with adding more noise to the training data (not label) may be helpful.

Don't argue about this by just saying if you disagree with these hypothesis. It will be more meaningful to discuss with experiments to verify them, no matter the results prove them right, or prove them wrong.

  • 1
    $\begingroup$ have this same issue as OP, and we are experiencing scenario 1. $\endgroup$
    – pangyuteng
    Commented Jan 16, 2020 at 18:03

From Ankur's answer, it seems to me that:

Accuracy measures the percentage correctness of the prediction i.e. $\frac{correct-classes}{total-classes}$


Loss actually tracks the inverse-confidence (for want of a better word) of the prediction. A high Loss score indicates that, even when the model is making good predictions, it is $less$ sure of the predictions it is making...and vice-versa.


High Validation Accuracy + High Loss Score vs High Training Accuracy + Low Loss Score suggest that the model may be over-fitting on the training data.


A model can overfit to cross entropy loss without over overfitting to accuracy.

There is a key difference between the two types of loss:

  1. Accuracy measures whether you get the prediction right
  2. Cross entropy measures how confident you are about a prediction

For example, if an image of a cat is passed into two models. Model A predicts {cat: 0.9, dog: 0.1} and model B predicts {cat: 0.6, dog: 0.4}. Both model will score the same accuracy, but model A will have a lower loss.

Because of this the model will try to be more and more confident to minimize loss. It works fine in training stage, but in validation stage it will perform poorly in term of loss. For example, for some borderline images, being confident e.g. {cat: 0.9, dog: 0.1} will give higher loss than being uncertain e.g. {cat: 0.6, dog: 0.4}

In short, cross entropy loss measures the calibration of a model. Mis-calibration is a common issue to modern neuronal networks. They tend to be over-confident. On Calibration of Modern Neural Networks talks about it in great details.


Let's say a label is horse and a prediction is:

cat   (25%)
dog   (35%)
horse (40%)

So, your model is predicting correct, but it's less sure about it. This is how you get high accuracy and high loss


Not the answer you're looking for? Browse other questions tagged or ask your own question.