1
$\begingroup$

Assume that we have some count data $x_{1}, \dots, x_{n}$, which take values $\{1, \dots, m\}$ and we have some estimator of the probability mass function, $\hat{\mathbf{p}} = (\hat{p}_{1}, \dots, \hat{p}_{m})$. In order to evaluate the performance of the estimator one can use Brier score which is the following. Let $\mathbf{I}_{i} = (0, \dots, 1, \dots, 0)$ is a vector in $\mathbb{R}^{m}$, with value $1$ is at the $t$-th position from the beginning of the vector, if $x_{i} = t$, for $t \in \{1, \dots, m\}$ and all $i=1,\dots, n$ . The Brier score is given by $$ BS = \frac{1}{n}\sum_{i=1}^{n}||\mathbf{I}_{i} - \hat{\mathbf{p}}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{n} \sum_{j=1}^{m}(I_{i,j} - \hat{p}_{j})^2. $$

I have got the following questions: what is the intuition behind this score? Will BS converge (in some sense) to the smallest value if (iff?) the estimator is consistent?

Is this basically a quadratic score, suggested by Stone in

M. Stone (1974) Cross-Validation and Multinomial Prediction Biometrika, Vol. 61, pp. 509-515 ?

How common is the use of it in practice?

$\endgroup$
2
$\begingroup$

The simplest way for me to think about the Brier score is to think of it as the equivalent of mean-square error for this type of task. It is a particular proper scoring rule, with the property:

If a cost is levied in proportion to a proper scoring rule, the minimal expected cost corresponds to reporting the true set of probabilities.

It thus favors good calibration of the probability model, rather than properties like accuracy that are based on an assumed value of a particular probability cutoff to make assignments of categories.

In practice I suspect that log-loss is more frequently used as a proper scoring rule, because that's what underlies logistic regression. The Brier score is easily applied to evaluating any type of probability model, however, and you will find it frequently recommended here as a good way to discriminate among models. Today you would find 236 posts containing "Brier score" on this site, close to the 246 containing "log-loss".

$\endgroup$
4
  • 4
    $\begingroup$ This summary comes from the Stata manuals, which are on-line. (I am not plagiarising, because I wrote it.) Glenn Wilson Brier (1913–1998) was an American meteorological statistician who, after obtaining degrees in physics and statistics, was for many years head of meteorological statistics at the U.S. Weather Bureau in Washington, DC. [...] Brier worked especially on verification and evaluation of predictions and forecasts, statistical decision making, the statistical theory of turbulence, the analysis of weather modification experiments, and the application of permutation techniques $\endgroup$
    – Nick Cox
    Aug 14 '20 at 14:10
  • $\begingroup$ The key idea was introduced in 1950. Logistic regression had been invented then, but was a very long way from being routine or easy to apply. $\endgroup$
    – Nick Cox
    Aug 14 '20 at 14:11
  • $\begingroup$ @NickCox Harrell has mentioned how weather forecasting has for years used probabilities instead of discrete outcomes ("90% chance of rain" vs "It's gonna rain"). It's interesting that Brier was a meteorologist. $\endgroup$
    – Dave
    Aug 14 '20 at 14:53
  • 1
    $\begingroup$ Meterologists and climatologists are very accustomed to dealing with uncertainty. In essence, climate is a statistical idea. Long before Robert Heinlein the geographer A.J. Herbertson wrote "Climate is what on an average we may expect, weather is what we actually get." . $\endgroup$
    – Nick Cox
    Aug 14 '20 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.