# ARIMA model selection with out of sample testing

I am trying to select an ARIMA model for a time series using out of sample testing (similar to how you would do it for any machine learning algorithm). I divide my data set into a training and test set and for each ARIMA model:

1. Fit the model to the training set
2. If the model has a good enough AIC or BIC move to step (3)
3. Forecast $|test set|$ steps ahead
4. Use a statistic to determine the goodness of forecasting for the test set
5. Select the model with the best results from (3)

The issue I have arises from (3). Naturally, the longer the forecast is the worse it gets. For a large enough sample, $|test set|$ can get pretty large as well. As a result, (3) might not be representative of how long I'd "keep" the model before refitting to account for recent changes.

To make this solution more robust, should I:

1. predict $n$ steps ahead
2. compare it to the next $n$ results in the test set using say MAE
3. Store this MAE in an array
4. Refit the model with the addition of the $n$ seen last time
5. Goto (1)

and then at the end, average all the of the MAEs stored in (3) to get a final "score".

Or is my original method also sufficient?

## migrated from quant.stackexchange.comDec 17 '16 at 13:03

This question came from our site for finance professionals and academics.

My preferred strategy is to conduct a computer-based experiment where for each of the L time series in your study:

1. Specify the # of periods in your forecast horizon , say NF

2. Specify the # of recent origins that you wish to consider where forecasts get launched from , say K origins

3. Specify M different modelling options , say M=10 for example

e.g.

1. simple one-step AIC/BIC ARIMA model & parameter selection ignoring deterministic structure

2. iterative utilization of acf/pacf to form an ARIMA model that separates signal from noise as suggested by Box and Jenkins ignoring possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses

3. ARIMA model/parameter selection like step 2 plus including possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses

4. Model/parameter selection including possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses excluding ARIMA structure

5. Same as 3 but allow parameters and error variance to change over time

6. Same as 4 but allow parameters and error variance to change over time