2
$\begingroup$

I am seeking advice for the best way to score a multi-output/multitask classification model's output.

Problem setup

A simplified version of the model is as follows:

  • Training data have F features, say they are descriptors of people (ethnicity, hair color, eye color, medical data)
  • Target variable is age

Obviously this is a regression problem, but just for fun I'd like to frame it as a classification problem using these bins:

  • Age <= 25,
  • 25 < Age <= 35,
  • 35 < Age <= 45, and
  • Age > 45

For each input "person" X, my model currently outputs probabilities of membership in each bin, that is: ( P(X <= 25yo), P(25yo < X <= 35), P(35yo < X <= 45yo), P(X > 45yo) ).

Main Question - How to score model output?

The low-hanging-fruit metric is calculating percent accuracy by finding the highest-probability age for each test-set person and comparing their binned real age, but I've read good arguments for using proper/strictly-proper scoring rules (e.g. Brier score/MSE, log scoring rule) instead of percent accuracy for binary classifiers from Frank Harrell and others here. I can't seem to find a classification scoring rule that generalizes to n-output/target variables, as there is in my my case where continuous output is discretized into mutually exclusive bins.

I could cook up some generalized metric by computing MSEs for each bin or something similar, but I figure I'd ask here for more informed minds if there's a standard approach here that I've overlooked.

Really appreciate any help, let me know if anything needs clarification.

$\endgroup$

2 Answers 2

4
$\begingroup$

There's some ambiguity (and I can't ask for clarification in the comments because I'm a new account).

I'd say first off all, in the scenario that you described, you're looking at an ordinal regression problem, not just a multi-class classification problem. That is, it's somewhere between a continuous regression task and a classification task. Accuracy is a bad measure, as you mention, because it 1) doesn't tell you whether you were close or not and 2) doesn't account for imbalance in the dataset.

Here is a paper that performs and ordinal regression task (specifically targetin age) in a neural network (CNN using faces as the inputs): Rank consistent ordinal regression for neural networks with application to age estimation, where they propose a metric they call CORAL (COnsistent RAnk Logits), specifically as their loss function for training.

Prior to that they talk about how it's common to break the task up into K binary classification tasks (i.e. >= x or < x). I'm not sure on the specifics of how their loss function differs from previous attempts since what I describe next is very similar. In essence they are calculating cross-entropy loss (more on this below) for K binary classifiers and weighting them by class frequency.

For this paper, they chose to use MAE and MSE both as measures of model performance. I recently did some ordinal regression on a task where the ranks happened to very nicely follow a Poisson distribution and found that mean Poisson deviance was a far better criterion than MSE or MAE. In classical ordinal logistic regression, some measures of pseudo-R^2 are also occasionally used (such as McFadens, see another comment on that here).

Now, if you're discussing a true multi-class classification problem (where there is no inherent ordering to the classes). You most likely want either weighted or regular categorical cross-entropy (weighted if you have imbalance during training).

For evaluating model performance of a multi-task classifier it depends on whether your classes are balanced or not, but generally I report both F1 score and accuracy as the major metrics and break precision/recall/accuracy out by class deeper in any reporting or examination of the model that I perform. Both F1 score and accuracy can be balanced in various ways. For F1 score I like to report both weighted and micro-averaged F1 score. For a model that performs really well on all classes, all of these measures will converge.

$\endgroup$
2
  • 2
    $\begingroup$ Good that you recognized the ordinal nature of the categorized outcomes (+1). You might want to see Chapters 13 and 14 of Frank Harrell's Regression Modeling Strategies and its links for the highly general applicability of ordinal regression. Don't forget that F1 score and accuracy have limitations, noted in the question and in another answer. $\endgroup$
    – EdM
    Jul 7, 2023 at 12:42
  • 1
    $\begingroup$ +1, the ordinal aspect is indeed relevant here. (Not sure about OP's "real" application, as far as I understand, "age" is just a stand-in here.) Per the links in my answer, I have major reservations about any KPI that relies on "hard" classifications, be it accuracy, precision, F1 or any others, and indeed, the OP is asking about probabilistic classifications and proper scoring rules. Unfortunately, I am not aware of any proper scoring rules for ordinal multiclass classifications... I'm sure there is a paper in there somewhere. $\endgroup$ Jul 7, 2023 at 12:59
1
$\begingroup$

You can extend the log loss to the multiclass case. As you write, accuracy is not a good idea. (I assume you know binning is not good practice, and are only using it here to get mutually exclusive bins for classification. You can also use the log score for continuous predictions.)

$\endgroup$
2
  • $\begingroup$ Thanks! I was just circling around to add that I found Brier's initial definition [1] for his score function, where he generalizes to the multiclass case. This is great to have two proper score functions, though. Yep I'm binning just as you say; however, the binning is in the target variable, not the input features. There is still information loss as with feature binning, but I'm finding it gets expressed in some really neat ways. Thanks again - will accept answer and close in a few hours. [1]: en.wikipedia.org/wiki/Brier_score#Original_definition_by_Brier $\endgroup$ Jul 4, 2023 at 22:55
  • 1
    $\begingroup$ OK - but note that the binning is problematic both for features and for outcomes. But that's a different question. $\endgroup$ Jul 5, 2023 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.