According to Wikipedia, the "original definition" of the Brier Score is:
$$BS=\frac{1}N\sum_{t=1}^N\sum_{i=1}^R(f_{ti}-o_{ti})^2 $$
Where $R$ is the number of classes, $N$ is the number of forecasting instances, $f_{ti}$ is the forecast probability of the $t$-th instance belonging to the $i$-th class, and $o_{ti}$ is the outcome (either $0$ or $1$).
I've got some data in which people predict whether the unemployment rate in the next quarter will be <2.5%, 2.5-5%, 5-7.5%, or >7.5%. So it's ordered categorical data. The subjects need to predict the probability the unemployment rate will fall into these categories, and their probabilities need to sum to 1. I've been encouraged to use the Brier Score to assess the performance of individual forecasters, but there's something that bothers me.
Consider Person 1:
Person 1 really had no clue about how to forecast unemployment. This person just assigned all four categories an equal probability, and ended up with a Brier Score of 0.06 + 0.06 + 0.06 + 0.56 = 0.75.
Then compare Person 2:
Person 2 had some knowledge suggesting that unemployment would be high. The correct category was ">7.5% unemployment" and Person 2 thought there was a .3 probability of that happening - thus Person 2 did better than Person 1 in this regard. Person 2 assigned a .7 probability to there being 5-7.5% unemployment.
The Brier Score for Person 2 is 0 + 0 + 0.49 + 0.49 = 0.98. So according to the Brier Score, Person 2 is worse than Person 1.
I find this very counterintuitive because Person 2 actually has some clue what they are doing, and furthermore assigned the correct category a higher probability than did Person 1 (0.3 vs 0.25).
Is this a problem in my particular case?
Assuming it is a problem in my particular case, would it have been OK if the categories were truly nominal as opposed to ordered categorical?
Assuming it is a problem in my case, what should I use instead of the Brier Score?