Assume that we have some count data $x_{1}, \dots, x_{n}$, generated by probability mass function $\textbf{p} = \{p_{1}, \dots, p_{s} \}$. Let $\hat{\theta}$ be some estimator of $\textbf{p}$.

In order to assess the estimator $\textbf{p}$, let us use Brier score, which is defined as $$ BS(\hat{\theta}) = \frac{1}{n}\sum_{i=1}^{n}||\mathbf{I}_{i} - \hat{\theta}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{n} \sum_{j=1}^{m}(I_{i,j} - \hat{\theta}_{j})^2, $$ where $\mathbf{I}_{i} = (0, \dots, 1, \dots, 0)$ is a vector in $\mathbb{R}^{s}$, with value $1$ at the $t$-th position from the beginning of the vector, if $x_{i} = t$, for $t \in \{1, \dots, s\}$ and all $i=1,\dots, n$.

The question is: in the sum above, do we need to exclude data point $x_{i}$ in the computation of $\hat{\theta}$, when we compute $||\mathbf{I}_{i} - \hat{\theta}||_{2}^{2}$? I mean, should we do it in the same way as we do when compute leave-one-out cross-validation criterion?

  • 1
    $\begingroup$ perhaps you can link to papers where you feel it is unclear. $\endgroup$
    – seanv507
    Feb 8 at 13:59

This solely depends on your model validation strategy.

  • If you are interested in performance on the model training data, you would use all observations in your training data.
  • If you are interested in performance on an independent test data set, the score would be calculated only on those.
  • If you assess your model quality with leave-one-out or some other form of cross-validation, then you would do similar.

Thus said, the calculation of the Brier score follows the same rules as e.g. a naive measure like accuracy or MSE.

  • $\begingroup$ Dear @Michael M , I have a feeling that whenever people use Brier score, they use all the observations in the training data. Mostly, in the papers they do not specify it. I thought there is a reason behind it... $\endgroup$
    – ABK
    Feb 8 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.