Intuition behind Brier score

Question

Assume that we have some count data $x_{1}, \dots, x_{n}$, which take values $\{1, \dots, m\}$ and we have some estimator of the probability mass function, $\hat{\mathbf{p}} = (\hat{p}_{1}, \dots, \hat{p}_{m})$. In order to evaluate the performance of the estimator one can use Brier score which is the following. Let $\mathbf{I}_{i} = (0, \dots, 1, \dots, 0)$ is a vector in $\mathbb{R}^{m}$, with value $1$ is at the $t$-th position from the beginning of the vector, if $x_{i} = t$, for $t \in \{1, \dots, m\}$ and all $i=1,\dots, n$ . The Brier score is given by $$ BS = \frac{1}{n}\sum_{i=1}^{n}||\mathbf{I}_{i} - \hat{\mathbf{p}}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{n} \sum_{j=1}^{m}(I_{i,j} - \hat{p}_{j})^2. $$

I have got the following questions: what is the intuition behind this score? Will BS converge (in some sense) to the smallest value if (iff?) the estimator is consistent?

Is this basically a quadratic score, suggested by Stone in

M. Stone (1974) Cross-Validation and Multinomial Prediction Biometrika, Vol. 61, pp. 509-515 ?

How common is the use of it in practice?

Answer 1

The simplest way for me to think about the Brier score is to think of it as the equivalent of mean-square error for this type of task. It is a particular proper scoring rule, with the property:

If a cost is levied in proportion to a proper scoring rule, the minimal expected cost corresponds to reporting the true set of probabilities.

It thus favors good calibration of the probability model, rather than properties like accuracy that are based on an assumed value of a particular probability cutoff to make assignments of categories.

In practice I suspect that log-loss is more frequently used as a proper scoring rule, because that's what underlies logistic regression. The Brier score is easily applied to evaluating any type of probability model, however, and you will find it frequently recommended here as a good way to discriminate among models. Today you would find 236 posts containing "Brier score" on this site, close to the 246 containing "log-loss".

Answer 2

To add a little to EdM's answer (+1), if you minimise the mean squared error for a regression problem, the optimal solution is for the model to estimate the conditional mean of the target distribution. For a classification problem, with 1-of-c coding, the conditional mean of the target distribution will be (for each output) the conditional probability of the corresponding class. This makes the reason for it's use as a calibration metric a little clearer (at least for me).