3
$\begingroup$

I am trying to select an ARIMA model for a time series using out of sample testing (similar to how you would do it for any machine learning algorithm). I divide my data set into a training and test set and for each ARIMA model:

  1. Fit the model to the training set
  2. If the model has a good enough AIC or BIC move to step (3)
  3. Forecast $|test set|$ steps ahead
  4. Use a statistic to determine the goodness of forecasting for the test set
  5. Select the model with the best results from (3)

The issue I have arises from (3). Naturally, the longer the forecast is the worse it gets. For a large enough sample, $|test set|$ can get pretty large as well. As a result, (3) might not be representative of how long I'd "keep" the model before refitting to account for recent changes.

To make this solution more robust, should I:

  1. predict $n$ steps ahead
  2. compare it to the next $n$ results in the test set using say MAE
  3. Store this MAE in an array
  4. Refit the model with the addition of the $n$ seen last time
  5. Goto (1)

and then at the end, average all the of the MAEs stored in (3) to get a final "score".

Or is my original method also sufficient?

$\endgroup$

migrated from quant.stackexchange.com Dec 17 '16 at 13:03

This question came from our site for finance professionals and academics.

2
$\begingroup$

My preferred strategy is to conduct a computer-based experiment where for each of the L time series in your study:

  1. Specify the # of periods in your forecast horizon , say NF

  2. Specify the # of recent origins that you wish to consider where forecasts get launched from , say K origins

  3. Specify M different modelling options , say M=10 for example

    e.g.

    1. simple one-step AIC/BIC ARIMA model & parameter selection ignoring deterministic structure

    2. iterative utilization of acf/pacf to form an ARIMA model that separates signal from noise as suggested by Box and Jenkins ignoring possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses

    3. ARIMA model/parameter selection like step 2 plus including possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses

    4. Model/parameter selection including possible deterministic structure like level shifts, local time trends , seasonal pulses , pulses excluding ARIMA structure

    5. Same as 3 but allow parameters and error variance to change over time

    6. Same as 4 but allow parameters and error variance to change over time

    7. Simple Holt-Winters additive model

    8. Same as 7 but allow pulses to be identified and included

    9. Holt-Winters multiplicative model

    10. Same as 9 but allow pulses to be identified and included

Others as needed filling out the M possible modeling approaches

Now for each of the K origins and each of the M modelling options/approaches compute an out-of-sample error statistic for the NF values that were withheld . Compute an overall error measure suggesting the best approach for any individual time series from the L time series that you are examining based upon KxM iterations. Compute a Weighted MAPE to identify the best approach of the M possible approaches.

If you wish (incorrectly in my opinion) to limit your selection to ARIMA just use the second option from the list of M options.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy