I would like to use DM test for probabilistic forecasting case. My initial thinking was to give CRPS of two forecasting methods instead of raw forecast errors, where CRPS is calculated using prediction interval constructed under parametric assumption (Gaussian).
Let's say I have 100 observations (y)
and 100 point forecasts from method-1 (f1)
and 100 point forecasts from method-2 (f2)
. Hence, I have CRPS of 100 points from both method-1 and method-2, instead of raw errors of 100 points (y-f1
and y-f2
). In this setting, it seems DM test is working fine when I use CRPS, because I got meaningful results, but I could not find any article backing using CRPS or any other error metric (absolute error, square error, ...) instead of raw errors. If my understanding is correct, there are some papers using CRPS instead of raw errors, but these papers mostly appear in arxiv (probably not published in peer-reviewed journal yet). I also found an R package called SpecsVerification
. The function ScoreDiff
seems doing the same as the definition of the function is: Calculate the difference (mean score of the reference forecast) minus (mean score of the forecast). Uncertainty is assessed by the Diebold-Mariano test for equality of predictive accuracy.
Simply, my question is that can I use CRPS instead of raw errors in DM test? Or is there any alternatives to DM test that allows using CRPS?