Very balanced dataset and a multiclass classification problem, no context behind the inputs. Which evaluation metric to use?

Question

I have constructed a simple neural network model, for a classification problem, with 10 target classes where an input (with some number of features) is to be classified to only one of the 10 classes.

The input data that I am provided with is just features as numbers (and their target classes) - there is no context behind it, so nothing like "vitals of cancer patients" where certain incorrect classifications would need to be punished more than others. The final layer activation function is the softmax function, and the Loss function used is categorical cross entropy.

Also, the number of occurrences of each class are very similar to one another. It is very much a balanced dataset.

The evaluation metrics I currently consider are:

• The loss function itself: categorical cross entropy.
• Accuracy score (because of course)
• Confusion matrix

So my question is: Which evaluation metric is the most appropriate to use given no context whatsoever - just raw numbers and balanced classes?

To add on to this question - say we were adding contexts behind the data provided, which type of contexts would warrant using each of these metrics. What would be the common use cases and drawbacks of these metrics? Are there alternate metrics that are advisable to look at for this class of problems (classification problem with multiple classes)?

Answer 1

The trouble with accuracy is that your model does not predict discrete classes. The neural network outputs values on a continuum that have a (granted, weak) interpretation as the probabilities of class membership.

(I say it is weak because neural networks tend to lack probability calibration. That is, when they predict a probability of $p$, the event does not happen with probability $p$.)

Consequently, your model has $0\%$ accuracy. Every prediction is at least a little bit incorrect.

In order to get a positive accuracy score or a confusion matrix, you need to convert those continuous predictions into discrete categories, and this requires you to know the consequences of making incorrect decisions. I would argue that, if you do not know the consequences of the discrete decisions, you have no business making those decisions. All you should be doing is making accurate probability predictions. While you might be able to get an accurate model, even a high accuracy score could mislead someone into thinking your model does not make crucial mistakes. Mixing up two classes might be particularly disastrous, so much so that a user is willing to make sacrifices elsewhere (even leading to a less accurate model overall) in order to minimize how often such a mistake is made.

The standard way to assess the probability predictions is through the crossentropy loss (“log loss” in some circles, much of Cross Validated being one such circle), which you can normalize using McFadden’s $R^2$. Brier score, which you can normalize using Efron’s $R^2$, could be a useful measure, too.

Related Links

Why is accuracy not the best measure for assessing classification models?

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

Classification vs. Prediction