I am generating some random data to test out a function that calculates Brier and scaled Brier scores. See here for a reference (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0091249)
A scaled Brier score is supposed to take into account the baseline prevalence of the event, comparing some predictive model against a "null" model, here defined as just guess the mean prevalence. But what if some predictive model is even worse than the null model? Wouldn't that give a negative scaled Brier score, which is supposed to range from 0 to 1?
Perhaps I've confused some definitions. Any help much appreciated. Here is an example using R:
set.seed (1) x <- sample(0:1, 100, T) y <- sample(0:1, 100, T) brier <- function(pred, obs) mean((pred - obs)^2) briermax <- function(obs) mean((mean(obs) - obs)^2) brierscaled <- function(pred, obs) 1 - (brier(pred,obs) / briermax(obs)) # now calculate our results brier(x,y) ##  0.52 briermax(y) ##  0.2484 brierscaled(x,y) ##  -1.093398
The results for the Brier score seem appropriate, but the scaled score doesn't make sense. The Brier max is SMALLER (ie better) than the actual Brier, which is driving the negative result. Why? That is, couldn't one reasonable guess much worse than the mean, or some other null model, always making the max (i.e. worst) Brier score = 1?