5
$\begingroup$

We have trained two different neural networks for MNIST dataset. Here are the losses and accuracies obtained by these networks for the training data:

net0: loss: 20780.8291187
net1: loss: 209.928699374
net0: TRAIN ACCURACY     0.985890040888
net1: TRAIN ACCURACY     0.835298627336

The used loss function is the cross-entropy. We expect higher accuracies for lower losses, but here, the loss for net1 is about 100 times lower than net0 but its accuracy is lower. What is the reason?

$\endgroup$
7
  • 2
    $\begingroup$ stats.stackexchange.com/questions/256551/… $\endgroup$
    – Sycorax
    Jun 19, 2017 at 21:56
  • $\begingroup$ @Sycorax Thanks. So if cross-entropy is so fragile that it can be completely different from accuracy, why we rely on it so much? $\endgroup$
    – Hossein
    Jun 21, 2017 at 7:49
  • $\begingroup$ "Fragile" is the wrong word, to my mind. Cross entropy measures how well-calibrated a model is. Accuracy is scarcely informative because many poorly or well calibrated models can have the same accuracy. $\endgroup$
    – Sycorax
    Jun 21, 2017 at 14:14
  • 1
    $\begingroup$ It's not clear to me why that's a problem. Q: "Which model has better accuracy?" A: "net0" Q: "Which model has better cross-entropy?" A: "net1" Q: "Why?" A: "Accuracy and cross-entropy measure different things." $\endgroup$
    – Sycorax
    Jun 21, 2017 at 17:11
  • 1
    $\begingroup$ If all you care about is accuracy, a better accuracy is better. $\endgroup$
    – Sycorax
    Jun 21, 2017 at 21:27

1 Answer 1

4
$\begingroup$

Lower cost function error not means better accuracy. The error of the cost function represents how well your model is learning/able to learn with respect to your training examples.

Now the question is , Is the model learning something that I expect it to learn?

It can show very low learning curves error but when you actually testing the results (Accuracy) you are getting wrong detection's , this is called high variance.

The best is to monitor the both learning error and accuracy for each epoch/iteration, While cost function error goes down and accuracy goes up keep training, otherwise stop (:

The accuracy is not good enough? check : 1. do I have high variance problem ? add more training examples to generalize the learning (better find more examples which reminds the problems where your model fails on).

  1. Do I have high bias problem ? my model is pure or to complex , need to fix it / try something else.

http://www.holehouse.org/mlclass/10_Advice_for_applying_machine_learning.html

Good luck (:

$\endgroup$
1
  • $\begingroup$ you are missing the point here. Bias is measured on Train Set and Variance is measures on Validation set. Instead, the question is: why in my training set I have a small cross-entropy loss but bad accuracy. $\endgroup$
    – Seymour
    Mar 29, 2020 at 10:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.