3
$\begingroup$

I'm working in a scenario in which forecasters make probabilistic forecasts of events with binary outcomes. I know how to calculate a Brier score and can easily do that for any of my forecasters.

However, I notice that the Wikipedia page doesn't mention anything about Confidence Intervals, and googling produced various hits, but as far as I could tell no clear answer.

If it's not reasonable to put a confidence interval around an individual forecaster's Brier Score, I'd be interested to know why.

There's a related and unanswered question on this site here. My question is different as I'm mostly interested in the formula and some intuition behind it. However a description of how to do it in a program such as R would also be appreciated.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Bradley et AL. Recently published an article named "Sampling Uncertainty and Confidence Intervals for the Brier Score and Brier Skill Score". It has an analytical unbiased estimation of CI for the Brier score. $\endgroup$
    – Shahar Katz
    Apr 17, 2021 at 8:23
  • $\begingroup$ Yeah that's a great reference, but note their development of BSS is a bit unusual and assumes some constant reference score journals.ametsoc.org/view/journals/wefo/23/5/… $\endgroup$
    – Spätzle
    Jan 19, 2022 at 7:19

2 Answers 2

Reset to default
2
$\begingroup$

I am not 100% sure about my answer, but I thought about it myself before and here is some food for thought.

So brier score is just a mean of squared errors right? So if other assumptions are met, you can just treat it as any other mean and use CI for mean.

The problem is that in my experience squared errors are never normally distributed, so the parametric CI might not give you exact results. Another problem is if the errors are not independent, which happens often for forecasts. Of course, this only works if the brier score is computed in the test set.

Improve this answer
$\endgroup$
1
1
$\begingroup$

If your computing power permits, you can bootstrap (sample with replacement) your data to the same size for, say, 1000 times and compute the Brier score for each resulting bootstrapped data set. In this way you have a 1000-sample distribution of the Brier score. Then you can get whichever confidence interval you like.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.