0
$\begingroup$

I'm working on a multinomial classification model with approximately 3,000 distinct classes to predict in a single category. The model is a neural network and the loss is categorical cross-entropy.

One thing I've noticed is that when I change some hyper parameter or add/remove some input feature, the validation loss gets lower (better) while the accuracy goes higher. And sometimes it's the opposite.

These are examples of final values, when the model doesn't manage to improve the validation loss anymore. They are a bit contrived but not far from the reality.

conditions validation loss validation accuracy
feature A / hyperparam B 1.25 0.90
feature B / hyperparam A 1.05 0.85
feature C / hyperparam A 1.55 0.95

It is actually quite hard to find a combination of feature/params that improve both the loss and the accuracy in a significant manner. I must also say that overall the model is working pretty well, with the loss/acc metrics going smoothly as the training goes, and little overfitting.

I'm curious of what is the intuition behind the loss going worse while the accuracy improves (or reciprocally), for the same validation data, after changing a parameter or a feature. Is one of them the sign of a "better" model? Which one should I focus on ultimately?

Improve this question
$\endgroup$
1
  • $\begingroup$ An answer on another question mentions that this behaviour can happen when classes are imbalanced, in which case the model can get a better accuracy by simply predicting more of the dominant class, while the loss is more complex. I'm guessing this is a good candidate for explanation. Does that mean that we should rather focus on the loss than on the accuracy, then? $\endgroup$
    – Jivan
    Oct 17, 2021 at 17:37

1 Answer 1

Reset to default
3
$\begingroup$
  • We have many different metrics because each of them measures slightly different things. Nothing strange in such discrepancies.
  • Accuracy is a pretty poor metric and there are many issues with it.
  • Taking this aside, accuracy looks at hard classifications (zeros and ones), while metrics such as log-loss or squared error judge how the score predicted by model fits the data. Loss functions look at much more detailed and raw predictions as compared to “rounded” ones considered by accuracy.
  • You need to pick single metric you care about. If you have multiple metrics, sooner or later they will start diverging and it would be hard for you to choose priorities.
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the answer. I've recently switched to a different model architecture (transformers instead of convolutions) and for the exact same train/test dataset, the validation accuracy went consistently higher (better) across all epochs, while the loss went consistently much higher (worse) across all epochs. The new model seems to be "less confident" in his predictions, but give better predictions. Is that a good interpretation? $\endgroup$
    – Jivan
    Oct 18, 2021 at 10:58
  • $\begingroup$ @Jivan according to the loss, the model is worse, while according to accuracy, it is better. You cannot say that it "is" better. Moreover, as stated above, accuracy is a rather poor metric. $\endgroup$
    – Tim ♦
    Oct 18, 2021 at 13:18
  • $\begingroup$ well yes, I guess one can only say it’s better given a particular purpose — if this purpose is fine with better accuracy at the expense of loss then it’s probably fine, otherwise not. $\endgroup$
    – Jivan
    Oct 18, 2021 at 20:43
  • $\begingroup$ But I definitely see the point in the accuracy being a poor metric, especially in the case of unbalanced classes — this is actually the case here and I suspect that overall the model has effectively gone worse $\endgroup$
    – Jivan
    Oct 18, 2021 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.