1
$\begingroup$

I would like to empirically evaluate the performance of the AIC, BIC and Cross Validation as model selection criteria for time series forecasting, i.e. which one of these criteria leads to the best out of sample accuracy?

The experiment I want to run is the following:

  • I have 10 years of monthly data, split into 8 years of training data and 2 years of testing data.
  • For each possible model out of a family of models (for example set of $ARIMA(p,q,d)$ models), calculate the AIC, the BIC, the CV error and the out of sample error rate of the forecast on the test set. I will use time series cross validation (as in R's tsCV())
  • Check which one of the 3 criteria selected the best model in terms of out of sample error.

My question is:

  • Does this approach make sense?
  • Since for my out of sample error, I am generating a 24 step ahead forecast, should my CV be done using 1 step ahead forecasts or 24 steps ahead forecasts?
$\endgroup$
  • $\begingroup$ Do you fit, for example, ARIMA(p,d,q) model? Then you select the order and parameters using AIC, BIC and cross-validation? $\endgroup$ – Waqas Jul 21 '18 at 8:07
  • $\begingroup$ @Waqas yes, that's the idea. $\endgroup$ – Skander H. Jul 21 '18 at 19:03
0
$\begingroup$

I think you may do one (or both) of the following:

  1. On training data choose parameters ($\phi$ and $\theta$) and order ($p,d$ and $q$) of ARIMA model using AIC, BIC and cross-validation. Fix the chosen model and predict for $2$ years.

  2. On training data choose parameters ($\phi$ and $\theta$) and order ($p,d$ and $q$) of ARIMA model using AIC, BIC and cross-validation. Update the model (by AIC, BIC and cross-validation) after each prediction.

If you want to compare, you have to treat them all equally. In my experience, online learning techniques are better, when the size of the data is significant. Please see the link

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.