3
$\begingroup$

I am forecasting items and measuring the point forecast and distribution accuracy of numerous different models against actuals. To measure distribution accuracy I am using the continuous-ranked probability score (CRPS.)

If I only desire to know how part of my forecast distribution performed (all parts of the distribution above the median forecast), is there a way to do this?

Improve this question
$\endgroup$
6
  • $\begingroup$ Parts of this question are obscure. Could you explain what you mean by "CRPS"? What do you mean by "distribution accuracy ... about the 50th percentile"? $\endgroup$
    – whuber ♦
    Feb 22, 2021 at 20:45
  • $\begingroup$ Use quantile scores and average over the parts of the distribution you're interested in. See otexts.com/fpp3/distaccuracy.html for further explanation. $\endgroup$
    – Rob Hyndman
    Feb 22, 2021 at 21:24
  • $\begingroup$ @whuber: The CRPS is the Continuous Ranked Probability Score, a very common proper scoring rule for numerical density forecasts. $\endgroup$
    – Stephan Kolassa
    Feb 23, 2021 at 6:35
  • $\begingroup$ @RobHyndman thank you for sharing. This textbook is very useful. $\endgroup$
    – bonddr
    Feb 23, 2021 at 17:49
  • $\begingroup$ @RobHyndman if I am interested in the upper bound of the quantile range (51st - 99th), is it enough to take the average of the quantile score for those specific probs? I am using the fabletools::quantile_score function for each point. $\endgroup$
    – bonddr
    Feb 23, 2021 at 19:41

1 Answer 1

Reset to default
1
$\begingroup$

You can use quantile weighting for your Continuous Ranked Probability Score (CRPS). Gneiting & Ranjan, "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules", Journal of Business & Economic Statistics (2011) is precisely what you need.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This is very helpful. Thank you! $\endgroup$
    – bonddr
    Feb 23, 2021 at 17:47
  • $\begingroup$ @Stephan Kolassa are you aware of the codes for Quantile Weighted Scoreing Rules in R or MATLAB? $\endgroup$
    – Rollo99
    May 25, 2021 at 13:05
  • 1
    $\begingroup$ @Rollo99: unfortunately, I don't know of any implementations as such, but if I remember correctly, the formulas in Gneiting & Ranjan are not that hard. For R, you might find something in the scoringRules or the scoringutils packages (which I personally haven't used, so I can't be any more helpful; I just found them by searching on CRAN for "Gneiting"), Good luck! $\endgroup$
    – Stephan Kolassa
    May 25, 2021 at 13:20
  • $\begingroup$ thanks a lot for answering and for your help! $\endgroup$
    – Rollo99
    May 25, 2021 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.