# Intuition behind Brier score

Assume that we have some count data $$x_{1}, \dots, x_{n}$$, which take values $$\{1, \dots, m\}$$ and we have some estimator of the probability mass function, $$\hat{\mathbf{p}} = (\hat{p}_{1}, \dots, \hat{p}_{m})$$. In order to evaluate the performance of the estimator one can use Brier score which is the following. Let $$\mathbf{I}_{i} = (0, \dots, 1, \dots, 0)$$ is a vector in $$\mathbb{R}^{m}$$, with value $$1$$ is at the $$t$$-th position from the beginning of the vector, if $$x_{i} = t$$, for $$t \in \{1, \dots, m\}$$ and all $$i=1,\dots, n$$ . The Brier score is given by $$BS = \frac{1}{n}\sum_{i=1}^{n}||\mathbf{I}_{i} - \hat{\mathbf{p}}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{n} \sum_{j=1}^{m}(I_{i,j} - \hat{p}_{j})^2.$$

I have got the following questions: what is the intuition behind this score? Will BS converge (in some sense) to the smallest value if (iff?) the estimator is consistent?

Is this basically a quadratic score, suggested by Stone in

M. Stone (1974) Cross-Validation and Multinomial Prediction Biometrika, Vol. 61, pp. 509-515 ?

How common is the use of it in practice?

The simplest way for me to think about the Brier score is to think of it as the equivalent of mean-square error for this type of task. It is a particular proper scoring rule, with the property:

If a cost is levied in proportion to a proper scoring rule, the minimal expected cost corresponds to reporting the true set of probabilities.

It thus favors good calibration of the probability model, rather than properties like accuracy that are based on an assumed value of a particular probability cutoff to make assignments of categories.

In practice I suspect that log-loss is more frequently used as a proper scoring rule, because that's what underlies logistic regression. The Brier score is easily applied to evaluating any type of probability model, however, and you will find it frequently recommended here as a good way to discriminate among models. Today you would find 236 posts containing "Brier score" on this site, close to the 246 containing "log-loss".

• This summary comes from the Stata manuals, which are on-line. (I am not plagiarising, because I wrote it.) Glenn Wilson Brier (1913–1998) was an American meteorological statistician who, after obtaining degrees in physics and statistics, was for many years head of meteorological statistics at the U.S. Weather Bureau in Washington, DC. [...] Brier worked especially on verification and evaluation of predictions and forecasts, statistical decision making, the statistical theory of turbulence, the analysis of weather modification experiments, and the application of permutation techniques Aug 14, 2020 at 14:10
• The key idea was introduced in 1950. Logistic regression had been invented then, but was a very long way from being routine or easy to apply. Aug 14, 2020 at 14:11
• @NickCox Harrell has mentioned how weather forecasting has for years used probabilities instead of discrete outcomes ("90% chance of rain" vs "It's gonna rain"). It's interesting that Brier was a meteorologist.
– Dave
Aug 14, 2020 at 14:53
• Meterologists and climatologists are very accustomed to dealing with uncertainty. In essence, climate is a statistical idea. Long before Robert Heinlein the geographer A.J. Herbertson wrote "Climate is what on an average we may expect, weather is what we actually get." . Aug 14, 2020 at 15:08

To add a little to EdM's answer (+1), if you minimise the mean squared error for a regression problem, the optimal solution is for the model to estimate the conditional mean of the target distribution. For a classification problem, with 1-of-c coding, the conditional mean of the target distribution will be (for each output) the conditional probability of the corresponding class. This makes the reason for it's use as a calibration metric a little clearer (at least for me).