up vote 4 down vote favorite
3

Suppose I make a bunch of probabilistic forecasts like:

  • 70% probability that sales growth will be 10-15% in Q1, 10% probability that sales growth will be > 15%, 20% probability that sales growth will be < 10%

Given the actual data, what's the best way to measure or track my accuracy? Brier score?

And can I average my Brier score for different types of forecasts? (e.g. Find the brier score for the prediction "there is 80% chance of rain" and average it with the sales growth forecast)

share|improve this question
1  
I'd hesitate to use the Brier score for ordinal outcomes with 3+ categories such as here where sales can be classified as low/medium/high. The Brier score treats each outcome as equidistant from the others. – RobertF Apr 27 '16 at 19:44
1  
Is your data inherently ordinal, as @RobertF appears to assume? If so, this adds complexity, as he writes, and it would be good if you could edit this into your post. If not, you can use proper scoring rules, like the Brier or others. And yes, you can average them. – Stephan Kolassa Apr 27 '16 at 20:12
    
I believe my example uses ordinal data, but I don't understand what you mean by "inherently ordinal". However, I could change my forecast to be just sales growth of 12%. If so I would use something like MAE? But the reason I included the probabilities is to take account of variability and rare events. For example, the expected sales growth of my forecast could be 12%, but there could be a small probability of negative sales growth. While unlikely to happen it may be valuable to know the likelihood of negative sales growth. So I want to reflect that in my forecast somehow. – Emile Apr 28 '16 at 1:47
    
By "inherently ordinal", I meant whether your underlying problem is ordinal, or whether the example you used just happened to be ordinal. Apparently, it's the former. – Stephan Kolassa Apr 28 '16 at 8:42
up vote 1 down vote

Your comment sounds as if you are really looking for a density forecast rather than a point forecast, i.e., you want to forecast the full probability distribution of the future outcome(s). This is a very good idea. Density forecasting is common in financial or econometric forecasting, but unfortunately it is rarely treated in other forecasting textbooks and courses. Tay & Wallis (2000, Journal of Forecasting) give a useful early survey.

The most common way of evaluating density forecasts uses the Probability Integral Transform (PIT). The canonical reference is Diebold, Gunther & Tay (1998, International Economic Review). Berkowitz (2001, Journal of Business & Economic Statistics) and Bao, Lee & Saltoglu (2007, Journal of Forecasting) give alternatives.

Recently, interest has risen in (proper) scoring rules, like the Brier score you mention. Literature includes Mitchell & Wallis (2011, Journal of Applied Econometrics) and Gneiting, Balabdaoui & Raftery (2007, JRSS-B).

Finally, Gneiting & Katzfuss (2014, Annual Review of Statistics and its Application) gives a more recent overview of density (or probabilistic) forecasting, focusing again on scoring rules.

share|improve this answer
    
Could you explain, in layman terms, the most appropriate method to make a density forecast and the most appropriate scoring rules for my example. Or you can point to an article/book. I guess what I'm asking is a recipe: steps to make a density/probabilistic forecast, then some other steps to evaluate accuracy of individual and/or multiple forecasts. And ideally I can do the calculation on a regular calculator (without specialized software). If some integral is required, what kind of approximations can I make to simplify the equation. Looking for something I can do "back of the envelope". – Emile Apr 29 '16 at 2:41
    
I would love to read those papers, but frankly they are over my head. – Emile Apr 29 '16 at 2:42
    
Ah. I'd recommend you look into a standard forecasting textbook, e.g., Ord & Fildes, Principles of Business Forecasting, section 5.2 and others, where prediction intervals are calculated using a normal distribution approach. Hyndman & Athanosopoulos unfortunately don't cover density forecasting. As to scoring rules, I don't really think there is a clear "best" one in your case - just pick one that you can implement easily. However, you will likely at least need normal distribution tables, so it would be good if you looked at R.... – Stephan Kolassa Apr 29 '16 at 10:30
    
... Hyndman & Athanasopoulos do everything with R, so their book serves as a nice introduction to forecasting using R. Good luck! – Stephan Kolassa Apr 29 '16 at 10:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.