4
$\begingroup$

When computing the mean squared error of a regression model, we get a metric in square units. For ease of interpretation, we can therefore instead compute the root mean squared error, which are in units of the target variable and also in a value range that is is closer to the actual values that we are predicting, and hence easier to reason about than the squared errors which can be several orders of magnitude different.

As I understand it, Brier score is a mean squared error for probabilistic classification outputs. Granted that probabilities don't have units, I would have expected it to still be more straightforward to report the probabilistic error as "root Brier score" as this would allow us to talk about the average probability deviation of the model instead of the average squared probability deviation. The latter not only seems less intuitive to me, but also means that it would bias towards lower numbers since squaring probabilities would make them smaller.

I noticed that there is something called a Brier Skill Score, which seems to be similar to a $R^2$ score in the sense that it is comparing the model towards a baseline model. I also found this post Name of mean absolute error analogue to Brier score? explaining the lack of an absolute version of the Brier score: it is not a proper scoring rule (although I don't fully understand the linked paper). It also makes sense that something like MAPE does not need to exist as probabilities are bounded 0-1. However, I could not find anything on a "root Brier score". I wonder if there is a fundamental reason for this, e.g. something related to the 0-1 probability space, that makes this variation of the Brier score redundant or if there are other reasons (maybe just the lack of units and that the scale distortion of squaring is not as large for probabilities?) that it is not used to the same extent as RMSE is used in regular regression problems.

$\endgroup$
1
  • 2
    $\begingroup$ The opening comment "the squared errors which can be several orders of magnitude larger" needs to add "or smaller", consistent with a point you make later. Or just say "different". $\endgroup$
    – Nick Cox
    Nov 18, 2023 at 11:07

1 Answer 1

2
$\begingroup$

One reason is that the "mean squared error" interpretation of a Brier score doesn't hold for Brier's more general form of the score, which can apply to more than two classes. If you use the original form for a binary outcome, the value is twice the "mean squared error." If you use the binary-outcome "mean squared error" form with more than two outcome classes, you no longer have a proper scoring rule.

$\endgroup$
1
  • 3
    $\begingroup$ I guess for the 2-class case one could compute the square root of the mean squared error; it’s just that it’s now standard practice to report the Brier score and its decompositions on the square scale. $\endgroup$ Nov 18, 2023 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.