Ideal scoring rules for multitask classification?

Question

I am seeking advice for the best way to score a multi-output/multitask classification model's output.

Problem setup

A simplified version of the model is as follows:

• Training data have F features, say they are descriptors of people (ethnicity, hair color, eye color, medical data)
• Target variable is age

Obviously this is a regression problem, but just for fun I'd like to frame it as a classification problem using these bins:

• Age <= 25,
• 25 < Age <= 35,
• 35 < Age <= 45, and
• Age > 45

For each input "person" X, my model currently outputs probabilities of membership in each bin, that is: ( P(X <= 25yo), P(25yo < X <= 35), P(35yo < X <= 45yo), P(X > 45yo) ).

Main Question - How to score model output?

The low-hanging-fruit metric is calculating percent accuracy by finding the highest-probability age for each test-set person and comparing their binned real age, but I've read good arguments for using proper/strictly-proper scoring rules (e.g. Brier score/MSE, log scoring rule) instead of percent accuracy for binary classifiers from Frank Harrell and others here. I can't seem to find a classification scoring rule that generalizes to n-output/target variables, as there is in my my case where continuous output is discretized into mutually exclusive bins.

I could cook up some generalized metric by computing MSEs for each bin or something similar, but I figure I'd ask here for more informed minds if there's a standard approach here that I've overlooked.

Really appreciate any help, let me know if anything needs clarification.

Answer 1

There's some ambiguity (and I can't ask for clarification in the comments because I'm a new account).

I'd say first off all, in the scenario that you described, you're looking at an ordinal regression problem, not just a multi-class classification problem. That is, it's somewhere between a continuous regression task and a classification task. Accuracy is a bad measure, as you mention, because it 1) doesn't tell you whether you were close or not and 2) doesn't account for imbalance in the dataset.

Here is a paper that performs and ordinal regression task (specifically targetin age) in a neural network (CNN using faces as the inputs): Rank consistent ordinal regression for neural networks with application to age estimation, where they propose a metric they call CORAL (COnsistent RAnk Logits), specifically as their loss function for training.

Prior to that they talk about how it's common to break the task up into K binary classification tasks (i.e. >= x or < x). I'm not sure on the specifics of how their loss function differs from previous attempts since what I describe next is very similar. In essence they are calculating cross-entropy loss (more on this below) for K binary classifiers and weighting them by class frequency.

For this paper, they chose to use MAE and MSE both as measures of model performance. I recently did some ordinal regression on a task where the ranks happened to very nicely follow a Poisson distribution and found that mean Poisson deviance was a far better criterion than MSE or MAE. In classical ordinal logistic regression, some measures of pseudo-R^2 are also occasionally used (such as McFadens, see another comment on that here).

Now, if you're discussing a true multi-class classification problem (where there is no inherent ordering to the classes). You most likely want either weighted or regular categorical cross-entropy (weighted if you have imbalance during training).

For evaluating model performance of a multi-task classifier it depends on whether your classes are balanced or not, but generally I report both F1 score and accuracy as the major metrics and break precision/recall/accuracy out by class deeper in any reporting or examination of the model that I perform. Both F1 score and accuracy can be balanced in various ways. For F1 score I like to report both weighted and micro-averaged F1 score. For a model that performs really well on all classes, all of these measures will converge.

Answer 2

You can extend the log loss to the multiclass case. As you write, accuracy is not a good idea. (I assume you know binning is not good practice, and are only using it here to get mutually exclusive bins for classification. You can also use the log score for continuous predictions.)