I would like to use DM test for probabilistic forecasting case. My initial thinking was to give CRPS of two forecasting methods instead of raw forecast errors, where CRPS is calculated using prediction interval constructed under parametric assumption (Gaussian).

Let's say I have 100 observations (y) and 100 point forecasts from method-1 (f1) and 100 point forecasts from method-2 (f2). Hence, I have CRPS of 100 points from both method-1 and method-2, instead of raw errors of 100 points (y-f1 and y-f2). In this setting, it seems DM test is working fine when I use CRPS, because I got meaningful results, but I could not find any article backing using CRPS or any other error metric (absolute error, square error, ...) instead of raw errors. If my understanding is correct, there are some papers using CRPS instead of raw errors, but these papers mostly appear in arxiv (probably not published in peer-reviewed journal yet). I also found an R package called SpecsVerification. The function ScoreDiff seems doing the same as the definition of the function is: Calculate the difference (mean score of the reference forecast) minus (mean score of the forecast). Uncertainty is assessed by the Diebold-Mariano test for equality of predictive accuracy.

Simply, my question is that can I use CRPS instead of raw errors in DM test? Or is there any alternatives to DM test that allows using CRPS?


Yes, you can.

The DM test only requires the loss differential to be covariance stationary, nothing more. There is nothing special about squared errors or absolute errors or CRPS (in particular, propriety is not even required).

See Diebold (2013), "Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests", for a thorough account of what the DM test assumes and what it actually means, and Gneiting & Katzfuss (2014), "Probabilistic Forecasting", for an actual (published, peer reviewed) use of DM test on CRPS.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.