0
$\begingroup$

I am building an ARIMA model and did a grid search to find which values to use for my AR and MA components using the AIC criteria (this was using all of my data). The results are in this graphic: enter image description here

Now I want to use cross-validation (as described here by Rob Hyndman) to select a model from a few candidates chosen from my grid search.

I have read in Elements of Statistical Learning (Hastie, Tibshirani and Friedman) a description of the wrong way to do cross validation and that I should perform the entire model selection process within each fold, but I am unsure if this approach is crossing the line, especially since it is with a time series model.

Improve this question
$\endgroup$
4
  • $\begingroup$ It is interesting that first you select some models by AIC and then want to select one of them using cross validation. I wonder where you got this idea. Regarding what to do in each fold (that is, in each of the rolling windows -- since this is a time series setting), I would keep the model order (p,q) fixed and reestimate model parameters, and predict one step ahead. I would then obtain a relevant performance measure, e.g. mean squared error, across all the folds. This measure would be associated with the particular ARMA order (p,q). Then you could pick the (p,q) that delivers the best MSE. $\endgroup$
    – Richard Hardy
    Commented Apr 21, 2015 at 19:05
  • $\begingroup$ My advice (in accordance with ESL) is to perform cross-validation at each point of the grid, and choose the model based on these results. You do not give details on the size of the problem but computationally-wise it shouldn't be a problem. $\endgroup$
    – Asterion
    Commented Apr 21, 2015 at 20:55
  • $\begingroup$ AIC and leave-one out crossvalidation are asymptotically equivalent I doubt that there would be any benefit in using AIC then using cross validation (CV) to select models. Why don't you simply use CV to select models ? It may be computationally expensive to do both. $\endgroup$
    – forecaster
    Commented Apr 21, 2015 at 21:05
  • $\begingroup$ when doing a grid search approach for a (p,d,q)(P,D,Q) problem is 6 dimensional problem, you would have a very computationally intensive search space. Have you had a chance to look at auto.arima page 8 in this article that uses a less computationally intensive methodology to search for optimal models ? $\endgroup$
    – forecaster
    Commented Apr 21, 2015 at 21:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.