Diebold Mariano test for non time-series data Can the Diebold Mariano test be used to compare the MSE of two models (trained on the same data) for non time series data?
 A: Yes, it can. There is nothing in the construction of the test that would make it unsuitable for non time series data. However, note the following subtlety: the test is for comparing forecasts, not models. As Diebold puts it in his "Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use
and Abuse of Diebold–Mariano Tests" (2015) paper (ungated version here),

The Diebold–Mariano (DM) test was intended for comparing forecasts; it has been, and remains, useful in that regard. The DM test was not intended for comparing models. Much of the large ensuing literature, however, uses DM-type tests for comparing models, in pseudo-out-of-sample environments. In that case, simpler yet more compelling full-sample model comparison procedures exist; they have been, and should continue to be, widely used. The hunch that pseudo-out-of-sample analysis is somehow the “only,” or “best,” or even necessarily a “good” way to provide insurance against in-sample overfitting in model comparisons proves largely false. On the other hand, pseudo-out-of-sample analysis remains useful for certain tasks, perhaps most notably for providing information about comparative predictive performance during particular historical episodes.

