I wish to compare two forecasters based on their historical performance (i.e. I want to determine who is better and by how much). The issue is that the two forecasters have performed a different number of forecasts in the past. There is some overlap.
So take for example:
- Forecaster A with 150 forecasts has a Brier score of 0.25
- Forecaster B with 50 forecasts has a Brier score of 0.2
- Both A and B have forecasted the same 30 events (i.e. the overlap).
I would like to determine which forecaster is better. I was thinking the following options:
- Do the comparison only on the Brier score of the overlap
- Weigh the Brier scores relative to the number of events
- Padding with prob=0.5 for non overlaps
Any other suggestions?