How to evaluate the results of a multilabel classifier using the predicted probabilities?

Question

I can use sklearn accuracy_score to evaluate de predicted values of my multilabel classifier.

But how can I evaluate the predicted probabilities obtained with predict_proba?

Answer 1

Good for you to look at the predicted probability values themselves. They contain useful information yet often get forgotten in favor of predicting the category with the highest probability, which can lead to some strange behavior. (I would argue that the sklearn predict_proba method should be named predict and predict should be named predict_argmax_category.)

Two common ways to assess the predicted probabilities are the log loss and the Brier score. The trouble is that they do not have obvious interpretations the way that accuracy does or even as mean squared error does in the regression setting.

To get a sense of how good your predictions are, you might consider comparing to a baseline level of performance, the most obvious being that of a model that always predicts the overall proportion, no matter the feature values. That is, if your three categories make up $70\%$, $20\%$, and $10\%$ of the observations, respectively, always predict the first with probability $0.7$, the second with probability $0.2$, and the third with probability $0.1$. When you consider such a model, you can calculate its log loss or Brier score as as “must-beat” level of performance. You can also do a quantification of how much you have improved upon such a level of performance by doing an $R^2$-style calculation:

$$ 1-\dfrac{ \text{ Performance of your model (log loss or Brier score) } }{ \text{ “Must-beat” performance (log loss or Brier, whichever is used above) } } $$

Such a calculation is related to the McFadden and Efron pseudo $R^2$ calculations describes on this helpful UCLA page. Interpreting if such a pseudo $R^2$ still has its difficulties and is not as straightforward as saying that $0.95$ is like an A in school that makes us happy while $0.5$ is like an F in school that makes us sad. However, it does give some context to a measure of performance that might otherwise be more difficult to interpret.

This is totally aligned with how Gneiting and Resin (2023) develop the $R^*$ that they call a universal coefficient of determination.

REFERENCE

Gneiting, Tilmann, and Johannes Resin. "Regression diagnostics meets forecast evaluation: Conditional calibration, reliability diagrams, and coefficient of determination." Electronic Journal of Statistics 17.2 (2023): 3226-3286.