How to measure probabilistic forecast accuracy?

Suppose I make a bunch of probabilistic forecasts like:

• 70% probability that sales growth will be 10-15% in Q1, 10% probability that sales growth will be > 15%, 20% probability that sales growth will be < 10%

Given the actual data, what's the best way to measure or track my accuracy? Brier score?

And can I average my Brier score for different types of forecasts? (e.g. Find the brier score for the prediction "there is 80% chance of rain" and average it with the sales growth forecast)

• I'd hesitate to use the Brier score for ordinal outcomes with 3+ categories such as here where sales can be classified as low/medium/high. The Brier score treats each outcome as equidistant from the others. Apr 27 '16 at 19:44
• Is your data inherently ordinal, as @RobertF appears to assume? If so, this adds complexity, as he writes, and it would be good if you could edit this into your post. If not, you can use proper scoring rules, like the Brier or others. And yes, you can average them. Apr 27 '16 at 20:12
• I believe my example uses ordinal data, but I don't understand what you mean by "inherently ordinal". However, I could change my forecast to be just sales growth of 12%. If so I would use something like MAE? But the reason I included the probabilities is to take account of variability and rare events. For example, the expected sales growth of my forecast could be 12%, but there could be a small probability of negative sales growth. While unlikely to happen it may be valuable to know the likelihood of negative sales growth. So I want to reflect that in my forecast somehow. Apr 28 '16 at 1:47
• By "inherently ordinal", I meant whether your underlying problem is ordinal, or whether the example you used just happened to be ordinal. Apparently, it's the former. Apr 28 '16 at 8:42