# Compare two forecasters on Brier score

I wish to compare two forecasters based on their historical performance (i.e. I want to determine who is better and by how much). The issue is that the two forecasters have performed a different number of forecasts in the past. There is some overlap.

So take for example:

• Forecaster A with 150 forecasts has a Brier score of 0.25
• Forecaster B with 50 forecasts has a Brier score of 0.2
• Both A and B have forecasted the same 30 events (i.e. the overlap).

I would like to determine which forecaster is better. I was thinking the following options:

1. Do the comparison only on the Brier score of the overlap
2. Weigh the Brier scores relative to the number of events
3. Padding with prob=0.5 for non overlaps

Any other suggestions?

• If someone makes 150 predictions and get 140 right, and another person makes 75 predictions and gets 30 of them right, I would say that the first person had stronger ability to predict. Would you not apply this same logic to Brier score?
– Dave
Dec 19, 2020 at 18:23
• No because it is probability-weighted prediction Dec 19, 2020 at 18:28
• What do you mean by a “probability-weighted decision”, and why does that make a difference?
– Dave
Dec 19, 2020 at 19:04
• The Brier score has a decision component (True/False) and a probability confidence component. So, compare the Brier of a method which is accurate 90% of the time but always provides a probability certainty of 51% on CORRECT decisions, with that of a method which is 80% accurate that gives 100% probability certainty on the CORRECT decisions. Dec 19, 2020 at 19:15
• The point of Brier score is that it only considers the class probabilities without ever making a decision about predicted class membership, so it is not clear what you mean.
– Dave
Dec 19, 2020 at 19:39