Compare two forecasters on Brier score I wish to compare two forecasters based on their historical performance (i.e. I want to determine who is better and by how much).  The issue is that the two forecasters have performed a different number of forecasts in the past. There is some overlap.
So take for example:

*

*Forecaster A with 150 forecasts has a Brier score of 0.25

*Forecaster B with 50 forecasts has a Brier score of 0.2

*Both A and B have forecasted the same 30 events (i.e. the overlap).

I would like to determine which forecaster is better. I was thinking the following options:

*

*Do the comparison only on the Brier score of the overlap

*Weigh the Brier scores relative to the number of events

*Padding with prob=0.5 for non overlaps

Any other suggestions?
 A: Option 1 is the safest of the three listed.
One problem with option 2 is that it over-rewards obvious high-confidence predictions. Suppose that forecasters A and B each try to predict 10 coin tosses, and each get 5 right. Now suppose that A makes 100 obvious predictions like "I am 100% confident that the sun will not explode in the next hour" (maybe because A is trying to game your system). Then A's average Brier score over all predictions is much better than B's, even though A hasn't really demonstrated any additional ability.
Option 3 is also problematic in that it over-penalizes the forecaster for not making a prediction - if forecaster A predicts with 90% confidence that "France will not declare war on Italy this year", and forecaster B didn't make a prediction, it's unfair to assume that B would have only assigned a 50% probability to such an obvious event.
One possible way to do better would be to use information about how impressive forecasts are. Assigning 45% confidence to an event which is generally thought highly unlikely and actually happens should be much more impressive than assigning 55% confidence to an event which is generally thought highly likely and indeed happens. It would help to have access to historical odds data from bookies or prediction markets about the events that are being forecast.
