# About computation of Brier score

Assume that we have some count data $$x_{1}, \dots, x_{n}$$, generated by probability mass function $$\textbf{p} = \{p_{1}, \dots, p_{s} \}$$. Let $$\hat{\theta}$$ be some estimator of $$\textbf{p}$$.

In order to assess the estimator $$\textbf{p}$$, let us use Brier score, which is defined as $$BS(\hat{\theta}) = \frac{1}{n}\sum_{i=1}^{n}||\mathbf{I}_{i} - \hat{\theta}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{n} \sum_{j=1}^{m}(I_{i,j} - \hat{\theta}_{j})^2,$$ where $$\mathbf{I}_{i} = (0, \dots, 1, \dots, 0)$$ is a vector in $$\mathbb{R}^{s}$$, with value $$1$$ at the $$t$$-th position from the beginning of the vector, if $$x_{i} = t$$, for $$t \in \{1, \dots, s\}$$ and all $$i=1,\dots, n$$.

The question is: in the sum above, do we need to exclude data point $$x_{i}$$ in the computation of $$\hat{\theta}$$, when we compute $$||\mathbf{I}_{i} - \hat{\theta}||_{2}^{2}$$? I mean, should we do it in the same way as we do when compute leave-one-out cross-validation criterion?

• perhaps you can link to papers where you feel it is unclear. Feb 8 at 13:59