Can you overfit with proper scoring rules, e.g., Brier score?

Question

I have read a lot suggestions and literature about using Brier score to measure model performance. It seems to be likened to the holy grail of model evaluation metrics because it is a proper scoring rule that optimizes against the true probabilities per class directly. But does that mean achieving BS < 0.1 on training data mean the model is a good model or can it also be considered suspect due to overfitting?

Answer 1

Much as I like proper scoring rules, and I like them a lot, they are just as susceptible to overfitting as any other model performance measure.

Suppose we know that we are dealing with a particular class of data generating processes, e.g., in an OLS situation. Then we know that the predictive distribution will be a t distribution, and the two parameters we need to estimate are the conditional mean and the unconditional variance. "Vanilla" overfitting occurs when we fit noise in predicting the unconditional mean by mindlessly minimizing MSE.

But we can just as well fit noise in estimating the unconditional variance by minimizing a proper scoring rule!

And of course the same holds if we are dealing with conditional variance in a more general model, or higher moments, or nonparametric predictive distributions.

So, just as for guarding against "vanilla" overfitting, even if we assess entire distributions using proper scoring-rules, we always need to use a true holdout sample, and consider regularization.