I am trying to select an ARIMA model for a time series using out of sample testing (similar to how you would do it for any machine learning algorithm). I divide my data set into a training and test set and for each ARIMA model:
- Fit the model to the training set
- If the model has a good enough AIC or BIC move to step (3)
- Forecast $|test set|$ steps ahead
- Use a statistic to determine the goodness of forecasting for the test set
- Select the model with the best results from (3)
The issue I have arises from (3). Naturally, the longer the forecast is the worse it gets. For a large enough sample, $|test set|$ can get pretty large as well. As a result, (3) might not be representative of how long I'd "keep" the model before refitting to account for recent changes.
To make this solution more robust, should I:
- predict $n$ steps ahead
- compare it to the next $n$ results in the test set using say MAE
- Store this MAE in an array
- Refit the model with the addition of the $n$ seen last time
- Goto (1)
and then at the end, average all the of the MAEs stored in (3) to get a final "score".
Or is my original method also sufficient?