Can anyone explain why "classic" tests of forecast accuracy, (i.e. Diebold-Mariano test, Meese-Rogoff test and Morgan-Granger-Newbold test) are not suited for nested models?

I could not find a good explanation in literature.


The Diebold-Mariano test is well suited for the purpose it was designed for: to assess whether two forecasts produce loss of equal size in population. This holds regardless of the models that were used to generate these forecasts. (Actually, models are not needed; survey forecasts or market-based forecasts can be assessed as well.)

Diebold (2015) remarks:

The DM test was intended for comparing forecasts; it has been, and remains, useful in that regard. <...> For comparing forecasts, DM is the only game in town. There’s really nothing more to say.

The problems begin when the test is used to compare models rather than forecasts.

The DM test was not intended for comparing models. Unfortunately, however, much of the subsequent literature uses DM-type tests for comparing models, in pseudo-out-of-sample environments. In that case, simpler yet more compelling full-sample model comparison procedures exist; they have been, and should continue to be, widely used.

The problem with comparing models is that the main assumption that the loss differential be covariance stationary gets violated, resulting in null distributions of the test statistic that are not the same as the one of comparing forecasts. When this is ignored, things get sour.

For comparing models, with estimated parameters, the situation is more nuanced. DM-style tests are still indisputably relevant, but the issue arises as to appropriate critical values.

(Diebold (2015) gives a detailed explanation of what goes on there, but I will skip it since you seem to be interested in forecasts rather than models.)

But why is a comparison of (model-based) forecasts different from a comparison of models? Because there are two different questions being asked:

  1. Which forecast is better (not only in this particular sample, but in population)? vs.
  2. Which model is more likely to have generated the data?

You could have Model A that generated the data but that is difficult to estimate with high precision. Once estimated, Model A produces poor forecasts due to the estimation imprecision.
You could have Model B that did not generate the data but approximates the data quite well and is also easy to estimate with high precision. Once estimated, Model B delivers rather accurate forecasts (more accurate than those from Model A).
Forecasts generated by Model B is the answer to question 1, while Model A is the answer to question 2.

Edit: Elliott and Timmermann (2016) include a brief explanation of the case of nested models on p. 103-104 where the nested model may deliver inferior forecasts in practice but would deliver superior forecasts under perfect estimation precision (no error in estimating model parameters), which is a hypothetical setting.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.