How to evaluate multi-class classifier on probability prediction task?

Question

I have a balanced dataset where each object (song) has one of the four target class labels (mood of a song). Example:

ID	feature1	feture2	feature3	target_class
0	0.5	0.11	125	upbeat
1	0.23	0.75	136	sad

Dataset has some outliers which I decided not to remove since they aren't measurement errors. I want to predict probability of each object belonging to each of the four classes, for example:

ID	upbeat	sad	energetic	relaxing
0	0.75	0.13	0.5	0.7
1	0.2	0.65	0.03	0.12

I'm planning to use this classifier on the unknown data in order to build a music database. The reason why I want to predict probabilities is so that instead of choosing one of the four moods user will be able to choose in between of the four moods when searching for music in the database.

So I have a couple of questions:

1. How do I evaluate classifier in this case? I'm thinking LogLoss, ROC-AUC, Brier score and ECE. But I've read that ROC-AUC isn't a good metric when it comes to multiclass problem.While LogLoss is more informative in case of imbalanced dataset.
2. Is it correct to optimize LogLoss in order to tune hyperparameters?

Answer 1

What I suspect you have read is that classification accuracy becomes problematic in the case of imbalance. There are valid arguments that classification accuracy is problematic in the balanced setting, too. This link argues for so-called (strictly) proper scoring rules, whether the classes are imbalanced or not, of which log loss and Brier score are two examples. This question and the accepted answer discuss how other threshold-based metrics are problematic, whether there is balance or imbalance. Strictly proper scoring rules evaluate the raw probability predictions of models and can be thought of as seeking out the true (conditional) probability values.

Both log loss and Broer score can be normalized $R^2$-style as McFadden’s and Efron’s pseudo $R^2$, respectively, which may be easier to interpret as improvement over a simple benchmark.

Much of the discussion about log loss and Brier score is about the binary setting, but both log loss and Brier score are strictly proper in the multi-class setting, too. Either if these, perhaps normalized to a pseudo $R^2$, are likely to work well for you.

(In Keras lingo, log loss is categorical cross-entropy.)