# Diebold-Mariano test for evaluating 40 different forecast horizons

I have a question about the Diebold-Mariano test. I have different forecasting horizons (n-ahead = 1 to 40) and different forecasting origins (26). I.e. I employ a rolling origin evaluation approach. I want to compare the forecasting results of two empirical models. Now, should I:

1. Compare all n-ahead=1 forecasting errors for the two models, set h=1, and perform the test. Then, I would do the same for n-ahead=2 forecasts, but set h=2, and so on.
2. Alternatively, is it more common to compare the average errors over all forecasting origins? However, this would result in information loss, wouldn't it?

The answer depends on what forecasts you actually want to compare.

• If you are interested in each of the 40 horizons $$h=1,\dots,40$$, you could do 40 separate Diebold-Mariano tests. E.g. each horizon's forecast leads to a separate decision, and consequences of each decision can be identified using a loss function defined on the forecast error. (The loss functions could be different for each horizon.) Thus you would get a specific answer for each horizon.
You may want to adjust the significance level to account for multiple testing.
You may also consider treating this as a test of a joint hypothesis.
• If you treat the 40 horizons as one multivariate forecast, you could do a single test. E.g. there is a single decision based on the 40 forecasts and you can define a loss function on the multivariate forecast error such as $$L(e_1,\dots,e_{40})=\sum_{h=1}^{40} w_h|e_h|$$ with nonnegative weights $$w_h$$ for $$h=1,\dots,40$$; here $$e_h$$ for $$h=1,\dots,40$$ are forecast errors. Thus you would get a single answer covering all the horizons.

The choice between these approaches should be guided by what you really are interested in from the subject-matter perspective.

• Thanks a lot! I am interested in the former. However, I am still not sure how to deal with the different forecasting origins. Can I just pool them over the forecasting horizons? And what would be the correct denominator then? Should I still include the forecasting horizon in the denominator (set h=1, h=2, etc.), or can I discard this and set h=1 for all forecasting horizons, as I am not comparing them over all forecasting horizons? Commented Apr 21, 2023 at 7:36
• @Aneconomist, if you want to do 40 tests, you would specify the horizon separately for each test. If you want to do a single test, I think you would have to specify h=40, since this is the overlap of at least some forecast errors that you will have when rolling the window one period at a time. Commented Apr 21, 2023 at 8:32
• Thanks. Regarding the former statement "if you want to do 40 tests, you would specify the horizon separately for each test," this turns out to be a problem since the number of interactions is 26 and h goes up to 40, so we have h > n, which results in issues (k <- ((n + 1 - 2 * h + (h/n) * (h - 1))/n)^(1/2)). Is there a standard solution for this problem? Commented Apr 21, 2023 at 8:45
• @Aneconomist, the argument h=... is used for estimating the long-run variance of the loss differential. Instead of using the dm.test in R, you could take the loss differential series, estimate the long-run variance manually and then do a $t$-test using this estimate. There are probably many functions that estimate long-run variance based on Newey-West, Andrews or other methods automatically, without relying on h. (They select the lag length according to some sensible rules.) Commented Apr 21, 2023 at 9:14
• @Aneconomist, yes. The only function of h is for determining the lag length. However, when h is large, the standard way o using it for that may no longer be optimal. (E.g. trying to obtain autocorrelation at lag 40 from a time series of length 26 breaks down.) Commented Apr 21, 2023 at 9:50