I would like to empirically evaluate the performance of the AIC, BIC and Cross Validation as model selection criteria for time series forecasting, i.e. which one of these criteria leads to the best out of sample accuracy?
The experiment I want to run is the following:
- I have 10 years of monthly data, split into 8 years of training data and 2 years of testing data.
- For each possible model out of a family of models (for example set of $ARIMA(p,q,d)$ models), calculate the AIC, the BIC, the CV error and the out of sample error rate of the forecast on the test set. I will use time series cross validation (as in R's tsCV())
- Check which one of the 3 criteria selected the best model in terms of out of sample error.
My question is:
- Does this approach make sense?
- Since for my out of sample error, I am generating a 24 step ahead forecast, should my CV be done using 1 step ahead forecasts or 24 steps ahead forecasts?