I have built two models, one ARIMAX and one VAR, to compare against a baseline ARIMA model to predict a weekly economic time series of interest. I am primarily comparing the accuracy of my models using the RMSE of out of sample forecasts, but I also want to use the Diebold-Mariano test to confirm that the VAR model is better than the baseline and the ARIMAX, and that the ARIMAX does not perform better than the baseline.

I created rolling forecasts from 2008-2018. For each year's forecast I respecified the models - I selected a new lag level for the VAR model (based off of the HQ criterion from VARselect in R), and I selected new ARIMAX parameters (based off of auto.arima). My question is: should I show the DM Test results year by year (in 2008 the p-value of the test was ___ allowing us to reject the null hypothesis, in 2009 the p-value of the test was ...) or simply show the results of the DM Test when I group the forecasts together and run the test across the 11 year time period of forecasts. Should I report both?

For my VAR model, which based off of the RMSE I believe performs quite well across the time period, the DM Tests on the annual forecasts only suggest that the VAR model outperforms the baseline 4 out of 11 times (at a p=0.05 significance). When I group the forecasts together and look at the whole time series from 2008-2018, the results suggest that the VAR model outperforms the baseline at a p=0.01 level of significance.

Or does my methodology of re-specifying my models annually appear flawed from the beginning?

For what it's worth, my rolling forecasts produce a nowcast, a one-week-ahead forecast, and a two-weeks-ahead forecast (I conduct separate DM Tests as well as calculate separate RMSE measurements for the different horizons). This is short-term forecasting, so in the context in which this model would be used re-specifying lag-order frequently is not a problem.

  • 1
    $\begingroup$ How are you computing the DM test statistic at a single year? Are you averaging over horizons? Or do you have higher frequency data and you're averaging say 12 monthly 1-month ahead forecasts? Or some other way? Typically, the DM test is computed from a sample of forecasts at a fixed horizon over multiple origins. Also, the DM test does not compare models, it compares forecasts. $\endgroup$ – Chris Haug Jan 2 at 17:47
  • $\begingroup$ Sorry, I should have included that originally. I am using weekly data, so I conducted the DM Test for each year of 52 observations. I run separate tests for the nowcast, one-week-ahead, and two-weeks-ahead forecasts. Point well-taken regarding the purpose of the DM test -- I need to be more careful with the language there, as I understand you're entirely correct about it comparing the forecasts rather than the models. $\endgroup$ – laborEcon Jan 2 at 18:43

First off, yes, doing a rolling weekly forecast and identifying VAR or ARIMA orders on a yearly basis only is certainly an admissible approach. You can fix orders and parameters for all time, or fix orders on a yearly basis and refit the parameters weekly, or change both of them every week, etc. These are all admissible as long as you base the choice of orders and parameters only on data available at that time.

Whether your approach will actually work better than the others depends entirely on the data. Do you have subject matter knowledge that suggests that the process might change at a fixed point in time every year and, if so, when that would be? Is there some physical or administrative/legal reason why you feel that say, on January 1st, you would do well to allow for a change in model specification? If not, you might want to consider allowing for the model to change over time in a less restrictive way (for example, time-varying parameter VAR models, regime switching models, by testing for structural changes over time and finding new orders when you find one, and so on).

Note that there are also other reasons than raw forecast performance for refitting models less often: it costs money to produce forecasts (in terms of computing equipment, time spent, hiring consultants, etc).

As for your DM testing procedure: the arbitrary splitting of the data at calendar year boundaries is a substantive decision. As you've noted, by splitting the data like this, you seem to obtain opposite conclusions (whole dataset says that your forecast is better, year by year seems less positive). You obviously can't choose one or the other based on which one says what you want it to say. Unless the choice of split is meaningful in terms of the application domain (or in terms of the evaluation itself, e.g. if my end of year bonus depends on the result of the DM test over that year), it's hard to justify without looking like you're trying to manipulate the results.

Overall, most people would want to know if the process which generates forecasts is superior to some benchmark process; the fact that you change model orders on a yearly basis is encapsulated in that process and does not necessarily imply that it must be evaluated separately year by year. Evaluating it year by year also gives you less information: it's possible to lose to the benchmark most years but still be better overall if the losses are much smaller than the wins.

In general, something that's more useful to look at in terms of the relative performance of two forecasts over time is a plot of the cumulative loss differential:

$$CLD_{t+h}^{(h)} = \sum_{s=1}^t \left[L(\hat{Y}^{F1}_{s+h|1:s},Y_{s+h})-L(\hat{Y}^{F2}_{s+h|1:s},Y_{s+h})\right]$$

$L(x,y)$ is an arbitrary loss function, say $L(x,y) = (x-y)^2$ for example. A jump up is a single instance where forecast F2 outperforms F1, and an upward trend represents consistent outperformance of F2 over F1 over a period of time. This doesn't involve any arbitrary splitting up of the data, and shows the relative performance at every point in time. It will help you avoid a forecast which is better overall in the past 10 years but stopped working recently (which is probably where you need it to work well the most...).

| cite | improve this answer | |
  • $\begingroup$ +1. Cumulative loss differential might be difficult to work with; in the simplest case of i.i.d. losses, it is a random walk. Meanwhile, raw losses are just an i.i.d. sequence for which using a Diebold-Mariano test (or some other test) would be simpler. The case of non i.i.d.'ness does not make cumulative loss easier to work with either, I think. It can be useful for visualization, though, as your last sentence can be understood to imply. $\endgroup$ – Richard Hardy Jan 3 at 5:39
  • $\begingroup$ This is incredibly helpful, thank you for taking the time to give such a thorough answer. First, you're right, re-specifying lag order on Jan. 1 is a pretty arbitrary choice. I'm going to spend some time today experimenting with varying it at more frequent intervals, say quarterly, monthly, and weekly, and see to what extent that can improve my forecasts. I will also, probably in future developments of this project, investigate your suggested alternative models and a procedure for testing for structural changes. Finally, thank you for the clarification regarding my use of the DM Test. $\endgroup$ – laborEcon Jan 3 at 14:02
  • $\begingroup$ What you say about it makes a lot of sense. I will also look into plotting out a CLD in R. $\endgroup$ – laborEcon Jan 3 at 14:03
  • $\begingroup$ @RichardHardy You're right, I wasn't suggesting doing any formal hypothesis tests on the cumulative loss difference, just plotting it, which is useful in its own right. The reason I use the cumulative sum is because it's easier to read as a plot than the loss difference, especially when you have many forecasts that you are comparing to the same benchmark and when their performance is only slightly better on average at each point. You do have to be careful when $h >1$ since even in the best case the loss differences won't be independent so you will get some less meaningful short term trending. $\endgroup$ – Chris Haug Jan 3 at 15:15
  • $\begingroup$ @ChrisHaug, great, we are on the same page. $\endgroup$ – Richard Hardy Jan 3 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.