time series Diebold-Mariano Test n-ahead forecast in R

Question

I have a question about "h" in dm.test & DM.test from package forecast & multDM, assume I set a ARIMA model to forecast n-ahead = 20 (using dynamic regression not one step forecast), so taht when I use DM.test which "h" should I define?

I have read this article , seems when the n-step forecast should choose h = n^1/3 + 1 (I'm not understand about this), where n is sample = 20 ; h = 4. Then want to Know in DM.test why can't choose h = 20 ? (the output is "NA")

I copy the data from this article

Actual <- c(1.22884,2.6684,3.41773,2.2392,2.12256,0.4638,-0.55081,1.18295,-2.4133,0.97947,0.55088,1.22792,-0.92351,-0.09028,1.68379,-0.61077,1.28104,-0.92225,-0.57811,0.7687)
fore1 <- c(0.902837,2.4492678,3.2075581,2.4383221,2.7751086,0.5931617,0.1085186,0.8785177,-1.165313,0.5937193,-0.003627,0.9943153,0.5194248,0.285099,0.5713786,0.2233359,0.327581,0.0846889,-0.083991,-0.073104)
fore2 <- c(0.8945434,2.3213521,2.5208157,1.908075,0.9507821,-0.610665,-1.11545,1.1116309,-2.777648,1.51728,0.4897679,1.047002,-1.344792,0.0019235,1.7465681,-1.063168,1.2719837,-1.289334,-0.464421,0.9785314)
DM.test(f1=fore1,f2=fore2,y=Actual,loss="SE",h=4,c=FALSE,H1="same")

statistic = 1.0302, forecast horizon = 4, p-value = 0.3029 (very close)

It's close to the article anwser, but when I using h = 20, the output return "NA"

statistic = NaN, forecast horizon = 20, p-value = NA

even h = 13

statistic = NaN, forecast horizon = 13, p-value = NA

What does the "h" mean?

I have confused about example dm.test. The example uses 80% as the training model, 20% as the test model with one-step-forecast, sets "h" = 1; however, if I use it in prediction n.step-forecast , what kind of "h" should I select?

one-step-forecast

f1 <- ets(WWWusage[1:80])
f2 <- auto.arima(WWWusage[1:80])
f1.out <- ets(WWWusage[81:100],model=f1)
f2.out <- Arima(WWWusage[81:100],model=f2)
accuracy(f1.out)
accuracy(f2.out)
dm.test(residuals(f1.out),residuals(f2.out),h=1) 

n-step-forecast

f1 <- ets(WWWusage[1:80])
f2 <- auto.arima(WWWusage[1:80])
f1.out <- forecast.ets(f1,h=20)
f2.out <- forecast(f2,h=20)
dm.test(WWWusage[81:100]-f1.out$mean , WWWusage[81:100]-f2.out$mean , h= 20)

DM = 0, Forecast horizon = 20, Loss function power = 2, p-value = 1

what is rule to select h when the example "forecast(f2,h=20)"?

Answer 1

I am not sure what you mean by

assume I set a ARIMA model to forecast n-ahead = 20 (using dynamic regression not one step forecast)

In any case, the role of $h$ is as follows:

• at time $t$ you predict $y_{t+h}$,
• at time $t+1$ you predict $y_{(t+1)+h}$,
• ...,
• at time $t+s$ you predict $y_{(t+s)+h}$.

Note that the forecast horizon is constant and equal to $h$ in each case.* Then you

• wait until the (end of) time period $(t+s)+h$,
• obtain the realized values $y_{t+h},\dots,y_{(t+s)+h}$,
• compare the forecasts to the realized values and
• collect the forecast errors.

You do this for two or more methods. Then you can test whether the expected value of forecast loss from the first forecast equals the expected value of the forecast loss of the second (or maybe several more) forecasts using the Diebold-Mariano test. In such a situation, you set h=h in the function options.

You are having a technical difficulty with setting $h=20$ in your particular example because the sample is too short for comparing forecasts of such long horizons. The method tries to estimate the long-run variance of the mean loss differential based on $20$ lags, and a sample of only $20$ observations is too short for that. Meanwhile, it works OK for values of $h$ up to 12, since you can (although with questionable accuracy) estimate the long-run variance using $12$ lags from a sample of $20$ observations.

*This is in contrast to predicting $1,\dots,h$ steps ahead from a fixed origin at time $t$. The Diebold-Mariano test is not suited for such situations.