I am using Brier score as my scoring metric, as opposed to log loss. My reason for doing so was because I read it was more focused on the positive class than log loss. But how is it?



The full quote from your link is:

The benefit of the Brier score is that it is focused on the positive class, which for imbalanced classification is the minority class. This makes it more preferable than log loss, which is focused on the entire probability distribution.

The Brier score is calculated as the mean squared error between the expected probabilities for the positive class (e.g. 1.0) and the predicted probabilities. ...

BrierScore = 1/N * Sum i to N (yhat_i – y_i)^2

The formula is correct. However, the explanation above it is wrong.

The Brier score is not only calculated over positive class instances. Rather, it is calculated over all instances, positive and negative. So if your outcomes can be $y=0$ or $y=1$, we can write the score as

$$ \frac{1}{N}\sum_{i=1}^n (\hat{y}_i-y_i)^2 = \frac{1}{N}\bigg(\sum_{y_i=0} \hat{y}_i^2 + \sum_{y_i=1} (\hat{y}_i-1)^2\bigg). $$

We see the contributions of both "negative" instances (with $y_i=0$, and larger $\hat{y}_i$ lead to higher loss) and "positive" instances (with $y_i=1$, and smaller $\hat{y}_i$ lead to higher loss).

(Note that $\hat{y}$ is assumed to be a probabilistic prediction, which your link does IMO not emphasize enough.)

As a matter of fact, the Brier score is a proper , just like the log loss. So both will be minimized in expectation by correct probabilistic predictions, i.e., when $P(y_i=1)=\hat{y}_i$. And if you use probabilistic predictions and proper scoring rules, all problems with unbalanced data vanish.

  • $\begingroup$ thanks, but why would you choose logloss over brier score? since it seems both are pretty much the same? $\endgroup$ – Maths12 Nov 4 '20 at 8:38
  • $\begingroup$ I wouldn't necessarily prefer log-loss over Brier. Why is LogLoss preferred over other proper scoring rules? discusses this question. $\endgroup$ – Stephan Kolassa Nov 4 '20 at 11:34
  • $\begingroup$ so it gives truer probabilities since it penalizes less when predictions are extreme? $\endgroup$ – Maths12 Nov 4 '20 at 16:10
  • 1
    $\begingroup$ Well, it compares predictions with the actual outcome. If the actual is $y=0$, then a prediction of $\hat{y}=0.3$ will be penalized less than a prediction of $\hat{y}=0.7$. Which is precisely what we want, because the first prediction gives a probability of 30% for the outcome to be $1$, and the second gives 70%, so the second one is wronger. $\endgroup$ – Stephan Kolassa Nov 4 '20 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.