Low loss and low accuracy. What is the reason? [duplicate]

We have trained two different neural networks for MNIST dataset. Here are the losses and accuracies obtained by these networks for the training data:

net0: loss: 20780.8291187
net1: loss: 209.928699374
net0: TRAIN ACCURACY     0.985890040888
net1: TRAIN ACCURACY     0.835298627336


The used loss function is the cross-entropy. We expect higher accuracies for lower losses, but here, the loss for net1 is about 100 times lower than net0 but its accuracy is lower. What is the reason?

• stats.stackexchange.com/questions/256551/…
– Sycorax
Jun 19, 2017 at 21:56
• @Sycorax Thanks. So if cross-entropy is so fragile that it can be completely different from accuracy, why we rely on it so much? Jun 21, 2017 at 7:49
• "Fragile" is the wrong word, to my mind. Cross entropy measures how well-calibrated a model is. Accuracy is scarcely informative because many poorly or well calibrated models can have the same accuracy.
– Sycorax
Jun 21, 2017 at 14:14
• It's not clear to me why that's a problem. Q: "Which model has better accuracy?" A: "net0" Q: "Which model has better cross-entropy?" A: "net1" Q: "Why?" A: "Accuracy and cross-entropy measure different things."
– Sycorax
Jun 21, 2017 at 17:11
• If all you care about is accuracy, a better accuracy is better.
– Sycorax
Jun 21, 2017 at 21:27

1 Answer

Lower cost function error not means better accuracy. The error of the cost function represents how well your model is learning/able to learn with respect to your training examples.

Now the question is , Is the model learning something that I expect it to learn?

It can show very low learning curves error but when you actually testing the results (Accuracy) you are getting wrong detection's , this is called high variance.

The best is to monitor the both learning error and accuracy for each epoch/iteration, While cost function error goes down and accuracy goes up keep training, otherwise stop (:

The accuracy is not good enough? check : 1. do I have high variance problem ? add more training examples to generalize the learning (better find more examples which reminds the problems where your model fails on).

1. Do I have high bias problem ? my model is pure or to complex , need to fix it / try something else.

http://www.holehouse.org/mlclass/10_Advice_for_applying_machine_learning.html

Good luck (:

• you are missing the point here. Bias is measured on Train Set and Variance is measures on Validation set. Instead, the question is: why in my training set I have a small cross-entropy loss but bad accuracy. Mar 29, 2020 at 10:08