Should I use the Diebold-Mariano test year-by-year or on the overall forecast? I have built two models, one ARIMAX and one VAR, to compare against a baseline ARIMA model to predict a weekly economic time series of interest. I am primarily comparing the accuracy of my models using the RMSE of out of sample forecasts, but I also want to use the Diebold-Mariano test to confirm that the VAR model is better than the baseline and the ARIMAX, and that the ARIMAX does not perform better than the baseline.
I created rolling forecasts from 2008-2018. For each year's forecast I respecified the models - I selected a new lag level for the VAR model (based off of the HQ criterion from VARselect in R), and I selected new ARIMAX parameters (based off of auto.arima). My question is: should I show the DM Test results year by year (in 2008 the p-value of the test was ___ allowing us to reject the null hypothesis, in 2009 the p-value of the test was ...) or simply show the results of the DM Test when I group the forecasts together and run the test across the 11 year time period of forecasts. Should I report both?
For my VAR model, which based off of the RMSE I believe performs quite well across the time period, the DM Tests on the annual forecasts only suggest that the VAR model outperforms the baseline 4 out of 11 times (at a p=0.05 significance). When I group the forecasts together and look at the whole time series from 2008-2018, the results suggest that the VAR model outperforms the baseline at a p=0.01 level of significance. 
Or does my methodology of re-specifying my models annually appear flawed from the beginning?
For what it's worth, my rolling forecasts produce a nowcast, a one-week-ahead forecast, and a two-weeks-ahead forecast (I conduct separate DM Tests as well as calculate separate RMSE measurements for the different horizons). This is short-term forecasting, so in the context in which this model would be used re-specifying lag-order frequently is not a problem. 
 A: First off, yes, doing a rolling weekly forecast and identifying VAR or ARIMA orders on a yearly basis only is certainly an admissible approach. You can fix orders and parameters for all time, or fix orders on a yearly basis and refit the parameters weekly, or change both of them every week, etc. These are all admissible as long as you base the choice of orders and parameters only on data available at that time. 
Whether your approach will actually work better than the others depends entirely on the data. Do you have subject matter knowledge that suggests that the process might change at a fixed point in time every year and, if so, when that would be? Is there some physical or administrative/legal reason why you feel that say, on January 1st, you would do well to allow for a change in model specification? If not, you might want to consider allowing for the model to change over time in a less restrictive way (for example, time-varying parameter VAR models, regime switching models, by testing for structural changes over time and finding new orders when you find one, and so on).
Note that there are also other reasons than raw forecast performance for refitting models less often: it costs money to produce forecasts (in terms of computing equipment, time spent, hiring consultants, etc).
As for your DM testing procedure: the arbitrary splitting of the data at calendar year boundaries is a substantive decision. As you've noted, by splitting the data like this, you seem to obtain opposite conclusions (whole dataset says that your forecast is better, year by year seems less positive). You obviously can't choose one or the other based on which one says what you want it to say. Unless the choice of split is meaningful in terms of the application domain (or in terms of the evaluation itself, e.g. if my end of year bonus depends on the result of the DM test over that year), it's hard to justify without looking like you're trying to manipulate the results.
Overall, most people would want to know if the process which generates forecasts is superior to some benchmark process; the fact that you change model orders on a yearly basis is encapsulated in that process and does not necessarily imply that it must be evaluated separately year by year. Evaluating it year by year also gives you less information: it's possible to lose to the benchmark most years but still be better overall if the losses are much smaller than the wins.
In general, something that's more useful to look at in terms of the relative performance of two forecasts over time is a plot of the cumulative loss differential:
$$CLD_{t+h}^{(h)} = \sum_{s=1}^t \left[L(\hat{Y}^{F1}_{s+h|1:s},Y_{s+h})-L(\hat{Y}^{F2}_{s+h|1:s},Y_{s+h})\right]$$
$L(x,y)$ is an arbitrary loss function, say $L(x,y) = (x-y)^2$ for example. A jump up is a single instance where forecast F2 outperforms F1, and an upward trend represents consistent outperformance of F2 over F1 over a period of time. This doesn't involve any arbitrary splitting up of the data, and shows the relative performance at every point in time. It will help you avoid a forecast which is better overall in the past 10 years but stopped working recently (which is probably where you need it to work well the most...).
