I have built two models, one ARIMAX and one VAR, to compare against a baseline ARIMA model to predict a weekly economic time series of interest. I am primarily comparing the accuracy of my models using the RMSE of out of sample forecasts, but I also want to use the Diebold-Mariano test to confirm that the VAR model is better than the baseline and the ARIMAX, and that the ARIMAX does not perform better than the baseline.
I created rolling forecasts from 2008-2018. For each year's forecast I respecified the models - I selected a new lag level for the VAR model (based off of the HQ criterion from VARselect in R), and I selected new ARIMAX parameters (based off of auto.arima). My question is: should I show the DM Test results year by year (in 2008 the p-value of the test was ___ allowing us to reject the null hypothesis, in 2009 the p-value of the test was ...) or simply show the results of the DM Test when I group the forecasts together and run the test across the 11 year time period of forecasts. Should I report both?
For my VAR model, which based off of the RMSE I believe performs quite well across the time period, the DM Tests on the annual forecasts only suggest that the VAR model outperforms the baseline 4 out of 11 times (at a p=0.05 significance). When I group the forecasts together and look at the whole time series from 2008-2018, the results suggest that the VAR model outperforms the baseline at a p=0.01 level of significance.
Or does my methodology of re-specifying my models annually appear flawed from the beginning?
For what it's worth, my rolling forecasts produce a nowcast, a one-week-ahead forecast, and a two-weeks-ahead forecast (I conduct separate DM Tests as well as calculate separate RMSE measurements for the different horizons). This is short-term forecasting, so in the context in which this model would be used re-specifying lag-order frequently is not a problem.