I am forecasting items and measuring the point forecast and distribution accuracy of numerous different models against actuals. To measure distribution accuracy I am using the continuous-ranked probability score (CRPS.)

If I only desire to know how part of my forecast distribution performed (all parts of the distribution above the median forecast), is there a way to do this?

  • $\begingroup$ Parts of this question are obscure. Could you explain what you mean by "CRPS"? What do you mean by "distribution accuracy ... about the 50th percentile"? $\endgroup$ – whuber Feb 22 at 20:45
  • $\begingroup$ Use quantile scores and average over the parts of the distribution you're interested in. See otexts.com/fpp3/distaccuracy.html for further explanation. $\endgroup$ – Rob Hyndman Feb 22 at 21:24
  • $\begingroup$ @whuber: The CRPS is the Continuous Ranked Probability Score, a very common proper scoring rule for numerical density forecasts. $\endgroup$ – Stephan Kolassa Feb 23 at 6:35
  • $\begingroup$ @RobHyndman thank you for sharing. This textbook is very useful. $\endgroup$ – bonddr Feb 23 at 17:49
  • $\begingroup$ @RobHyndman if I am interested in the upper bound of the quantile range (51st - 99th), is it enough to take the average of the quantile score for those specific probs? I am using the fabletools::quantile_score function for each point. $\endgroup$ – bonddr Feb 23 at 19:41

You can use quantile weighting for your Continuous Ranked Probability Score (CRPS). Gneiting & Ranjan, "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules", Journal of Business & Economic Statistics (2011) is precisely what you need.

  • $\begingroup$ This is very helpful. Thank you! $\endgroup$ – bonddr Feb 23 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.