I am forecasting items and measuring the point forecast and distribution accuracy of numerous different models against actuals. To measure distribution accuracy I am using the continuous-ranked probability score (CRPS.)

If I only desire to know how part of my forecast distribution performed (all parts of the distribution above the median forecast), is there a way to do this?

  • $\begingroup$ Parts of this question are obscure. Could you explain what you mean by "CRPS"? What do you mean by "distribution accuracy ... about the 50th percentile"? $\endgroup$ – whuber Feb 22 at 20:45
  • $\begingroup$ Use quantile scores and average over the parts of the distribution you're interested in. See otexts.com/fpp3/distaccuracy.html for further explanation. $\endgroup$ – Rob Hyndman Feb 22 at 21:24
  • $\begingroup$ @whuber: The CRPS is the Continuous Ranked Probability Score, a very common proper scoring rule for numerical density forecasts. $\endgroup$ – Stephan Kolassa Feb 23 at 6:35
  • $\begingroup$ @RobHyndman thank you for sharing. This textbook is very useful. $\endgroup$ – bonddr Feb 23 at 17:49
  • $\begingroup$ @RobHyndman if I am interested in the upper bound of the quantile range (51st - 99th), is it enough to take the average of the quantile score for those specific probs? I am using the fabletools::quantile_score function for each point. $\endgroup$ – bonddr Feb 23 at 19:41

You can use quantile weighting for your Continuous Ranked Probability Score (CRPS). Gneiting & Ranjan, "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules", Journal of Business & Economic Statistics (2011) is precisely what you need.

  • $\begingroup$ This is very helpful. Thank you! $\endgroup$ – bonddr Feb 23 at 17:47
  • $\begingroup$ @Stephan Kolassa are you aware of the codes for Quantile Weighted Scoreing Rules in R or MATLAB? $\endgroup$ – Rollo99 May 25 at 13:05
  • 1
    $\begingroup$ @Rollo99: unfortunately, I don't know of any implementations as such, but if I remember correctly, the formulas in Gneiting & Ranjan are not that hard. For R, you might find something in the scoringRules or the scoringutils packages (which I personally haven't used, so I can't be any more helpful; I just found them by searching on CRAN for "Gneiting"), Good luck! $\endgroup$ – Stephan Kolassa May 25 at 13:20
  • $\begingroup$ thanks a lot for answering and for your help! $\endgroup$ – Rollo99 May 25 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.