# Can you overfit with proper scoring rules, e.g., Brier score?

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring rule that optimizes against the true probabilities per class directly. But does that mean achieving BS < 0.1 on training data mean the model is a good model or can it also be considered suspect due to overfitting?

Much as I like proper scoring rules, and I like them a lot, they are just as susceptible to overfitting as any other model performance measure.

Suppose we know that we are dealing with a particular class of data generating processes, e.g., in an OLS situation. Then we know that the predictive distribution will be a t distribution, and the two parameters we need to estimate are the conditional mean and the unconditional variance. "Vanilla" overfitting occurs when we fit noise in predicting the unconditional mean by mindlessly minimizing MSE.

But we can just as well fit noise in estimating the unconditional variance by minimizing a proper scoring rule!

And of course the same holds if we are dealing with conditional variance in a more general model, or higher moments, or nonparametric predictive distributions.

So, just as for guarding against "vanilla" overfitting, even if we assess entire distributions using proper , we always need to use a true holdout sample, and consider regularization.

• sorry a bit late to game here, but say i calcaulte brier score on my test set, how can i tell it's over fitting? what is my measure, e.g. please see stackoverflow.com/questions/63355797/… Nov 7, 2020 at 21:25
• @Maths12: that's a reasonable question by itself. CrossValidated would be a better venue than SO. Unfortunately, the question you link is too old to migrate. I would recommend you re-post it here. Nov 8, 2020 at 7:25
• Nov 8, 2020 at 16:19