2
$\begingroup$

I am generating some random data to test out a function that calculates Brier and scaled Brier scores. See here for a reference (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0091249)

A scaled Brier score is supposed to take into account the baseline prevalence of the event, comparing some predictive model against a "null" model, here defined as just guess the mean prevalence. But what if some predictive model is even worse than the null model? Wouldn't that give a negative scaled Brier score, which is supposed to range from 0 to 1?

Perhaps I've confused some definitions. Any help much appreciated. Here is an example using R:

set.seed (1)
x <- sample(0:1, 100, T)
y <- sample(0:1, 100, T)

brier <- function(pred, obs) mean((pred - obs)^2)
briermax <- function(obs) mean((mean(obs) - obs)^2)
brierscaled <- function(pred, obs) 1 - (brier(pred,obs) / briermax(obs))

# now calculate our results
brier(x,y)
## [1] 0.52
briermax(y)
## [1] 0.2484
brierscaled(x,y)
## [1] -1.093398

The results for the Brier score seem appropriate, but the scaled score doesn't make sense. The Brier max is SMALLER (ie better) than the actual Brier, which is driving the negative result. Why? That is, couldn't one reasonable guess much worse than the mean, or some other null model, always making the max (i.e. worst) Brier score = 1?

$\endgroup$
1
$\begingroup$

I am not quite sure where the max Brier score originally was defined. I have looked at lots of references, but they all state the formula and no one explains where it came from (e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575184/).

I also encountered situations when model does much worse than always guessing answers with the constant probability (mean(p) in there reference above). And, yes, in that case you get a negative scaled Brier score.

I think the idea is to compare your model's Brier score with the Brier score from uninformative model (with const probability) and see how well it did. Of course, it's possible to do better and worse than Brier_max, which makes scaled brier score closer to 1 or negative respectively.

$\endgroup$
2
  • $\begingroup$ Thanks for your response @Susie. I think you identified the problem, which is that the scaled Brier score can be negative. This doesn't feel very scaled -- my intuition is that it should range from 0 to 1, but perhaps my intuition and expectations are wrong, which is why I am confused. $\endgroup$ – Gary Weissman May 13 '17 at 0:20
  • $\begingroup$ I think that setting scaled Brier's score to 0 in case the prediction is as good as random guess makes sense: it's like saying "The model is useless, since you would do equally well with random guess". $\endgroup$ – Federico Tedeschi Aug 13 '20 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.