ARIMA model selection with out of sample testing I am trying to select an ARIMA model for a time series using out of sample testing (similar to how you would do it for any machine learning algorithm). I divide my data set into a training and test set and for each ARIMA model:


*

*Fit the model to the training set

*If the model has a good enough AIC or BIC move to step (3)

*Forecast $|test set|$ steps ahead 

*Use a statistic to determine the goodness of forecasting for the test set

*Select the model with the best results from (3)


The issue I have arises from (3). Naturally, the longer the forecast is the worse it gets. For a large enough sample, $|test set|$ can get pretty large as well. As a result, (3) might not be representative of how long I'd "keep" the model before refitting to account for recent changes.
To make this solution more robust, should I:


*

*predict $n$ steps ahead

*compare it to the next $n$ results in the test set using say MAE

*Store this MAE in an array

*Refit the model with the addition of the $n$ seen last time

*Goto (1)


and then at the end, average all the of the MAEs stored in (3) to get a final "score".
Or is my original method also sufficient?
 A: My preferred strategy is to conduct a computer-based experiment where for each of the L time series in your study:


*

*Specify the # of periods in your forecast horizon , say NF

*Specify the # of recent  origins that you wish to consider where forecasts get launched from , say K origins

*Specify M different modelling options , say M=10 for example
e.g. 


*

*simple one-step AIC/BIC ARIMA model & parameter selection ignoring deterministic structure

*iterative utilization of acf/pacf to form an ARIMA model that separates signal from noise as suggested by Box and Jenkins ignoring possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses

*ARIMA model/parameter selection like step 2 plus including possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses

*Model/parameter selection including possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses excluding ARIMA structure

*Same as 3 but allow parameters and error variance to change over time

*Same as 4 but allow parameters and error variance to change over time 

*Simple Holt-Winters additive model 

*Same as 7 but allow pulses to be identified and included

*Holt-Winters multiplicative model

*Same as 9 but allow pulses to be identified and included
Others as needed filling out the M possible modeling approaches
Now for each of the K origins and each of the M modelling options/approaches compute an out-of-sample error statistic for the NF values that were withheld . Compute an overall error measure suggesting the best approach for any individual time series from the L time series that you are examining based upon KxM iterations. Compute a Weighted MAPE to identify the best approach of the M possible approaches.
If you wish (incorrectly in my opinion) to limit your selection to ARIMA just use the second option from the list of M options.
