5
$\begingroup$

I can use sklearn accuracy_score to evaluate de predicted values of my multilabel classifier.

But how can I evaluate the predicted probabilities obtained with predict_proba?

$\endgroup$

1 Answer 1

2
$\begingroup$

Good for you to look at the predicted probability values themselves. They contain useful information yet often get forgotten in favor of predicting the category with the highest probability, which can lead to some strange behavior. (I would argue that the sklearn predict_proba method should be named predict and predict should be named predict_argmax_category.)

Two common ways to assess the predicted probabilities are the log loss and the Brier score. The trouble is that they do not have obvious interpretations the way that accuracy does or even as mean squares error does in the regression setting.

To get a sense of how good your predictions are, you might consider comparing to a baseline level of performance, the most obvious being that of a model that always predicts the overall proportion, no matter the feature values. That is, if your three categories make up $70\%$, $20\%$, and $10\%$ of the observations, respectively, always predict the first with probability $0.7$, the second with probability $0.2$, and the third with probability $0.1$. When you consider such a model, you can calculate its log loss or Brier score as as “must-beat” level of performance. You can also do a quantification of how much you have improved upon such a level of performance by doing an $R^2$-style calculation:

$$ 1-\dfrac{ \text{ Performance of your model (log loss or Brier score) } }{ \text{ “Must-beat” performance (log loss or Brier, whichever is used above } } $$

Such a calculation is related to the McFadden and Efron pseudo $R^2$ calculations describes on this helpful UCLA page. Interpreting if such a pseudo $R^2$ still has its difficulties and is not as straightforward as saying that $0.95$ is like an A in school that makes us happy while $0.5$ is like an F in school that makes us sad. However, it does give some context to a measure of performance that might otherwise be more difficult to interpret.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.