Compare the predictive capacity of multivariate models

Question

I want to compare the prediction capacity of multivariate models (MGARCH, VAR, SVAR, VECM...). One option to do this is by comparing its error metrics: the mean square error or any other between the predicted values ​​and some test data.

However, I would also like to compare its predictive ability with hypothesis tests. In the univariate case there is the Diebold-Mariano test. There is even a multivariate version of this test, but it compares several predictions but not multivariate models.

I would like to know if you know of another test that allows this comparison between multivariate models. Or another method that allows to vectorize the prediction errors to compare the models.

Answer 1

If you want to rank the models (and then test whether the best model beats the other models), you will have to define a scalar performance metric (a scalar-valued loss function). With that in place, and since you have more than two models to compare, I refer you to the tag description of diebold-mariano-test:

In comparing multiple forecasts, we need to address the multiple comparisons problem. In such a case, the standard approach is the "multiple comparisons to the best" (MCB) test originally proposed by Koning et al. (2005). <...> It is rank-based, so it works with any accuracy measure. <...> A related alternative would be the Friedman-Nemenyi test (Demsar, 2006). Both the MCB and the Nememyi test are implemented in the TStools package for R.