1
$\begingroup$

I have two stationary time series, ts1 and ts2. I applied grangertest() in R and the result was that ts2 Granger causes ts1. I then applied an auto-regressive distributed lag model and also a auto-regressive model (removing ts2).

I performed a rolling forecast cross-validation to test the performance of the two models, and to my surprise, the AR model outperformed the ARDL model. How is this possible if the Granger test showed that the ARDL model is more valuable for forecasting?

I have plotted the MASE results for each of the cross-validation tests on this plot:enter image description here

and the reproducible R code is:

ts <- read.csv(url("https://raw.githubusercontent.com/allandf/r-funcs/master/data/MyTsData.csv"))

require(caret)
require(dLagM)

print(grangertest(ts1 ~ ts2, order=3,data = ts))
print(grangertest(ts2 ~ ts1, order=3,data = ts))

dataslice <- createTimeSlices(y = ts$ts1, initialWindow = 20, horizon = 1,                         
 fixedWindow = FALSE,skip = 0)
Q <- sum(abs(diff(ts$ts1)))/(length(ts$ts1)-1)

ARDLM_MAE <- 0
AR_MAE <- 0

for (j in 1:length(dataslice$train))
    {
    ARDLmodel <- ardlDlm(x = ts$ts2[unlist(dataslice$train[j])],y=ts$ts1[unlist(dataslice$train[j])],p=3,q=3,show.summary=FALSE)
    ARmodel <- ar(x = ts$ts1[unlist(dataslice$train[j])], aic = FALSE, order.max = 3)

    ARDLMpred <- ardlDlmForecast(ARDLmodel,x = ts$ts2[unlist(dataslice$test[j])])
    ARpred <- predict(ARmodel)

    ARDLM_MAE[j] <- abs(ARDLMpred$forecasts-ts$ts1[unlist(dataslice$test[j])])
    AR_MAE[j] <- abs(ARpred$pred[1]-ts$ts1[unlist(dataslice$test[j])])
}

ARDLM_MASE <- ARDLM_MAE / Q
AR_MASE <- AR_MAE / Q

mean(ARDLM_MASE)
mean(AR_MASE)

plot(AR_MASE,type='l',col='red',xlab="Cross-validation test number",ylab="MASE")
lines(1:56,ARDLM_MASE,type='l',col='green')
lines(0:56,(0:56)/(0:56),type='l')
legend(43,7.5,c("ARDLM", "ARM"),lty=c(1,1),lwd=c(2.5,2.5),col=c("green","red")) 
$\endgroup$
  • $\begingroup$ Why would you ever perform cross validation if in-sample statistical tests of significance were sufficient to determine the best model for forecasting? $\endgroup$ – Chris Haug Jul 20 '18 at 21:08
  • $\begingroup$ In sample statistical tests for model selection, and cross-validation to see how the models will perform in the real world? Are you implying that the reason why I am seeing a discrepancy in the model selection and performance is due to in-sample vs out of sample testing? Since the out of sample testing is done on data which was used for model selection, I wouldnt of expected the discrepancy surely? $\endgroup$ – Allan_ZA Jul 21 '18 at 19:33
1
$\begingroup$

Despite how some people tend to interpret it, Granger causality is not a test of predictive power, at all. What you have when you say "$X$ Granger-causes $Y$" is the fact that, given that the assumptions of the test are true, there is sufficient evidence that $X$ has an effect on $Y$. It does not at all mean that using $X$ instead of $Y$'s own lags only will provide for a better forecast (especially if the assumptions of the test are not true). For example, if the relationship between $X$ and $Y$ is harder to estimate (or less stable) than the relationship between $Y$ and its own lags.

In fact, using classic hypothesis tests for parameter significance in your model is not at all a good measure of whether those features should be included, in any context, and should not be your only means of model selection. That is the exact reason why cross validation exists: it answers the actual question you are asking, which is "what is the best model for prediction", and not another question like "is there evidence that $X$ has any impact on $Y$", whose answer is not actually that useful in answering the first question. The fact that they disagree is the reason why we have cross validation at all.

In short, no, Granger causality does not imply that using the causing variable will yield better forecasts, and it is normal for out-of-sample/cross validation model selection procedures to disagree with Granger causality, and you should always consider the results of OOS/CV first if what you are trying to do is select a model for prediction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.