# Can I measure the accuracy of a range of quantiles in my forecast distribution?

I am forecasting items and measuring the point forecast and distribution accuracy of numerous different models against actuals. To measure distribution accuracy I am using the continuous-ranked probability score (CRPS.)

If I only desire to know how part of my forecast distribution performed (all parts of the distribution above the median forecast), is there a way to do this?

• Parts of this question are obscure. Could you explain what you mean by "CRPS"? What do you mean by "distribution accuracy ... about the 50th percentile"? – whuber Feb 22 at 20:45
• Use quantile scores and average over the parts of the distribution you're interested in. See otexts.com/fpp3/distaccuracy.html for further explanation. – Rob Hyndman Feb 22 at 21:24
• @whuber: The CRPS is the Continuous Ranked Probability Score, a very common proper scoring rule for numerical density forecasts. – Stephan Kolassa Feb 23 at 6:35
• @RobHyndman thank you for sharing. This textbook is very useful. – bonddr Feb 23 at 17:49
• @RobHyndman if I am interested in the upper bound of the quantile range (51st - 99th), is it enough to take the average of the quantile score for those specific probs? I am using the fabletools::quantile_score function for each point. – bonddr Feb 23 at 19:41

• @Rollo99: unfortunately, I don't know of any implementations as such, but if I remember correctly, the formulas in Gneiting & Ranjan are not that hard. For R, you might find something in the scoringRules or the scoringutils packages (which I personally haven't used, so I can't be any more helpful; I just found them by searching on CRAN for "Gneiting"), Good luck! – Stephan Kolassa May 25 at 13:20