5
$\begingroup$

I wish to compare two forecasters based on their historical performance (i.e. I want to determine who is better and by how much). The issue is that the two forecasters have performed a different number of forecasts in the past. There is some overlap.

So take for example:

  • Forecaster A with 150 forecasts has a Brier score of 0.25
  • Forecaster B with 50 forecasts has a Brier score of 0.2
  • Both A and B have forecasted the same 30 events (i.e. the overlap).

I would like to determine which forecaster is better. I was thinking the following options:

  1. Do the comparison only on the Brier score of the overlap
  2. Weigh the Brier scores relative to the number of events
  3. Padding with prob=0.5 for non overlaps

Any other suggestions?

Improve this question
$\endgroup$
8
  • $\begingroup$ If someone makes 150 predictions and get 140 right, and another person makes 75 predictions and gets 30 of them right, I would say that the first person had stronger ability to predict. Would you not apply this same logic to Brier score? $\endgroup$
    – Dave
    Dec 19, 2020 at 18:23
  • 1
    $\begingroup$ No because it is probability-weighted prediction $\endgroup$
    – vzografos
    Dec 19, 2020 at 18:28
  • $\begingroup$ What do you mean by a “probability-weighted decision”, and why does that make a difference? $\endgroup$
    – Dave
    Dec 19, 2020 at 19:04
  • 1
    $\begingroup$ The Brier score has a decision component (True/False) and a probability confidence component. So, compare the Brier of a method which is accurate 90% of the time but always provides a probability certainty of 51% on CORRECT decisions, with that of a method which is 80% accurate that gives 100% probability certainty on the CORRECT decisions. $\endgroup$
    – vzografos
    Dec 19, 2020 at 19:15
  • $\begingroup$ The point of Brier score is that it only considers the class probabilities without ever making a decision about predicted class membership, so it is not clear what you mean. $\endgroup$
    – Dave
    Dec 19, 2020 at 19:39

1 Answer 1

Reset to default
3
$\begingroup$

Option 1 is the safest of the three listed.

One problem with option 2 is that it over-rewards obvious high-confidence predictions. Suppose that forecasters A and B each try to predict 10 coin tosses, and each get 5 right. Now suppose that A makes 100 obvious predictions like "I am 100% confident that the sun will not explode in the next hour" (maybe because A is trying to game your system). Then A's average Brier score over all predictions is much better than B's, even though A hasn't really demonstrated any additional ability.

Option 3 is also problematic in that it over-penalizes the forecaster for not making a prediction - if forecaster A predicts with 90% confidence that "France will not declare war on Italy this year", and forecaster B didn't make a prediction, it's unfair to assume that B would have only assigned a 50% probability to such an obvious event.

One possible way to do better would be to use information about how impressive forecasts are. Assigning 45% confidence to an event which is generally thought highly unlikely and actually happens should be much more impressive than assigning 55% confidence to an event which is generally thought highly likely and indeed happens. It would help to have access to historical odds data from bookies or prediction markets about the events that are being forecast.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yeah I am not so happy with either of these 3 options. Considering that option 1 can sometime mean very small overlap. Assuming that there is no ulterior motive from the forecasters (i.e. some AI bots) which of the 3 is the best option? Also, maybe there is another option I haven't considered? $\endgroup$
    – vzografos
    Dec 19, 2020 at 19:48
  • $\begingroup$ The problem with 2 is there even when there isn't ulterior motive, whenever one forecaster tends to be answering questions which are "easier" than those of the other forecaster - e.g. if A is mostly forecasting whether it will rain tomorrow, and B is forecasting results of close elections, B is unfairly handicapped. I guess this isn't an issue when the two forecasters are looking at rougly equally difficult questions - e.g. if A is forecasting whether it rains in 2009-10 and B is forecasting the same thing for 2011, a weighted average could be appropriate. $\endgroup$
    – fblundun
    Dec 19, 2020 at 20:01
  • $\begingroup$ I think you also have to consider when they are right (I know nothing of the brier score). If one was right decades ago, but another is right more recently then which do you really want to use? Structural breaks are not uncommon in time series, particularly when it is a long one in term of years. $\endgroup$
    – user54285
    Dec 21, 2020 at 23:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.