A test for superior predictive accuracy of one forecast over another.
The null hypothesis of the Diebold-Mariano test is that the expected forecast loss (e.g. expected mean squared forecast error) is equal for two alternative forecasts. In other words, its tests whether the two forecasts have the same accuracy in population.
The test makes use of the fact that asymptotically, the difference between the loss functions incurred by two forecasts (suitably standardized) approaches a normal distribution.
The original publication was Diebold & Mariano (1995, Journal of Business and Economic Statistics). Twenty years later, Diebold (2015, Journal of Business and Economic Statistics) offered some recollections on the test, its uses and abuses.
In comparing multiple forecasts, we need to address the multiple comparisons problem. In such a case, the standard approach is the "multiple comparisons to the best" (MCB) test originally proposed by Koning et al. (2005) for a re-analysis of the M3 forecasting competition. Most recently it has been applied to submissions in the M5 forecasting competition as well. It is rank-based, so it works with any accuracy measure (and appropriate point forecasts, Kolassa, 2020). A related alternative would be the Friedman-Nemenyi test (Demsar, 2006).
Both the MCB and the Nememyi test are implemented in the TStools package for R. An empirical comparison between the two is given by Hibon et al.'s 2012 ISF presentation.
