Choose best forecasting model doing backtesting with all the models

Question

My objective is to find the best ARIMA or Exponential Smoothing model to forecast one time serie. I know that, to choose the best model, one can make tests to know if the ts has trend, seasonality, is stationary, etc and then choose consequently the appropriate model.

But, what about doing backtesting with all possible models and choose the best one? i.e. the one that has better accuracy over time?

Let's put an example. Suppose that, for a given ts, we have available monthly data from Jan2016 to this last month (Oct2018) and we want to forecast the next November. The strategy is: fit all the ARIMA models (a representative subset) and all the Exponential Smoothing models for the last n months, calculate the accuracy and choose, for forecast November, the one with the best overall accuracy.

I mean, fit all the models with Jan16-Feb18 and forecast Mar18, calculate accuracy, then fit all the models with Jan16-Mar18 and forecast Apr18, calculate accuracy, etc.

With the actual power of the computers, this can be done in seconds.

What do you think?

Answer 1

Your overall approach is feasible, and essentially the same as time series cross validation. See this blog post by Rob Hyndman.

Note that in your case, since you plan on using ARIMA and ETS models, only the second part of the post applies to your approach (i.e. time series cross-validation). You can't use K-fold cross-validation.

A note of caution is that iterating thought all possible ARIMA models is unrealistic, you will need to put some bounds on the orders p,d,q and P,Q,D, additionally you will have to check for convergence for each set of parameters.

For ETS models however, the number of models is not an issue since it is fixed, and convergence isn't usually an issue either.

Answer 2

What you are proposing is to shoe-horn say L particular models while optimally estimating parameters for a given set of historical observations (N) and compute an out-of-sample mape/smape given a specified forecast horizon say K.

The problem with this is:

For any particular starting model(l) for any particular set of observations(n) for any particular forecast horizon (k) there my be pulses/step shifts/seasonal pulses/local time trends which are latent (waiting to be discovered via exploratory data analysis and identifiable thus requiring an altering in the model in a Baconian way (deductive). At the end a more correct/efficient model is dynamically formed which has and only has statistically significant and sufficient structure.

This scientific approach requires an iterative goal-seeking algorithm that tirelessly separates signal and noise. The heart of the matter is that to correctly evaluate a forecast for k periods one needs to iterate in a logical manner to optimally form a useful model before making the forecast. To lock in a named model is nieve as the optimized parameters can be needlessly be flawed by non-gaussian violations rendering the error process to be "colored" containing unexposed structure such as changes in error variance or parameters over time or untreated pulses/seasonal pulses,step-level shifts or local time trends .

At the end of the task one has L mapes or smapes for each of K origins. This leads directly to the task of determining a composite for each of the L starting models (candidates) and thus a declared "winner".