Is the Brier Score appropriate for ordered categorical data?

Question

According to Wikipedia, the "original definition" of the Brier Score is:

$$BS=\frac{1}N\sum_{t=1}^N\sum_{i=1}^R(f_{ti}-o_{ti})^2 $$

Where $R$ is the number of classes, $N$ is the number of forecasting instances, $f_{ti}$ is the forecast probability of the $t$-th instance belonging to the $i$-th class, and $o_{ti}$ is the outcome (either $0$ or $1$).

I've got some data in which people predict whether the unemployment rate in the next quarter will be <2.5%, 2.5-5%, 5-7.5%, or >7.5%. So it's ordered categorical data. The subjects need to predict the probability the unemployment rate will fall into these categories, and their probabilities need to sum to 1. I've been encouraged to use the Brier Score to assess the performance of individual forecasters, but there's something that bothers me.

Consider Person 1:

Person 1 really had no clue about how to forecast unemployment. This person just assigned all four categories an equal probability, and ended up with a Brier Score of 0.06 + 0.06 + 0.06 + 0.56 = 0.75.

Then compare Person 2:

Person 2 had some knowledge suggesting that unemployment would be high. The correct category was ">7.5% unemployment" and Person 2 thought there was a .3 probability of that happening - thus Person 2 did better than Person 1 in this regard. Person 2 assigned a .7 probability to there being 5-7.5% unemployment.

The Brier Score for Person 2 is 0 + 0 + 0.49 + 0.49 = 0.98. So according to the Brier Score, Person 2 is worse than Person 1.

I find this very counterintuitive because Person 2 actually has some clue what they are doing, and furthermore assigned the correct category a higher probability than did Person 1 (0.3 vs 0.25).

1. Is this a problem in my particular case?

2. Assuming it is a problem in my particular case, would it have been OK if the categories were truly nominal as opposed to ordered categorical?

3. Assuming it is a problem in my case, what should I use instead of the Brier Score?

Answer 1

1. In your case, the Brier score still "works" in that it incentivizes you to state the true forecast distribution (it is a proper scoring rule, see Gneiting and Raftery, 2007). For example, if the true probabilities were 0, 1/4, 1/4, 1/2, you would minimize your expected Brier score by actually stating these probabilities.
2. and 3. The Brier score is proper for both nominal and ordered data. However, as your example shows, it does not care about the forecast probabilities in the bins next to the one that realized. Whether this is problematic or not is largely a philosophical question. If you dislike this feature of the Brier score, you might use the Ranked Probability Score, defined as $$RPS = \sum_{i=1}^r BS(i),$$ where $BS(i)$ is the Brier score for the event that the outcome lies within the first $i$ categories (loosely quoting from the post linked above). In your example, Person 1 gets an RPS of $$RPS_1 = 0.25^2 + 0.5^2 + 0.75^2 + 0^2 = 0.875,$$ whereas Person 2 attains an RPS of $$RPS_2 = 0^2 + 0^2 + 0.7^2 + 0^2 = 0.49,$$ thus outperforming Person 1. Note that like the Brier score, the RPS is also a proper scoring rule (and thus justified from a stats theory perspective).