About computation of Brier score

Question

Assume that we have some count data $x_{1}, \dots, x_{n}$, generated by probability mass function $\textbf{p} = \{p_{1}, \dots, p_{s} \}$. Let $\hat{\theta}$ be some estimator of $\textbf{p}$.

In order to assess the estimator $\textbf{p}$, let us use Brier score, which is defined as $$ BS(\hat{\theta}) = \frac{1}{n}\sum_{i=1}^{n}||\mathbf{I}_{i} - \hat{\theta}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{n} \sum_{j=1}^{m}(I_{i,j} - \hat{\theta}_{j})^2, $$ where $\mathbf{I}_{i} = (0, \dots, 1, \dots, 0)$ is a vector in $\mathbb{R}^{s}$, with value $1$ at the $t$-th position from the beginning of the vector, if $x_{i} = t$, for $t \in \{1, \dots, s\}$ and all $i=1,\dots, n$.

The question is: in the sum above, do we need to exclude data point $x_{i}$ in the computation of $\hat{\theta}$, when we compute $||\mathbf{I}_{i} - \hat{\theta}||_{2}^{2}$? I mean, should we do it in the same way as we do when compute leave-one-out cross-validation criterion?

Answer 1

This solely depends on your model validation strategy.

• If you are interested in performance on the model training data, you would use all observations in your training data.
• If you are interested in performance on an independent test data set, the score would be calculated only on those.
• If you assess your model quality with leave-one-out or some other form of cross-validation, then you would do similar.

Thus said, the calculation of the Brier score follows the same rules as e.g. a naive measure like accuracy or MSE.