Rolling forecast accuracy is it preferred to re-select model on each iteration

Question

While measuring forecast accuracy using the rolling period approach:

• Should we re-calculate the model in each step. Ex: execute auto.arima() for each new window and calculate the error

OR

• Calculate the model on the initial window, use the parameters from this model to calculate accuracy at each step.

Rob Hyndman discusses both the approaches in his blog here - https://robjhyndman.com/hyndsight/rolling-forecasts/

However, I am not sure which is a preferred approach.

Answer 1

The purpose of pseudo-out-of-sample testing is to replicate real-life conditions in a controlled experiment. So, once you are done testing your models, what will you do when you actually have to produce real forecasts? Will you select a new model each time a new observation comes along? Or choose it once and then never touch it again? Or maybe at fixed intervals, or when a forecast comes out that "looks bad"?

Your pseudo-out-of-sample testing should replicate exactly the strategy that you will actually use once it's time to forecast for real. If you're going to change the model at every new observation in reality, then the errors you get in testing with a fixed model won't be representative, and vice-versa.

The question of which refitting/model selection frequency is best has no general answer and depends on your data. Refitting too often could lead to overfitting, and not often enough could miss real changes in the process. You would also have to consider the time it takes to select a model and the costs involved each time.

Answer 2

At any one origin ALL the known historical data should be used to form the best model and a set of parameters and a forecast. To assume that neither the best model has not changed or prior estimates of the best parameters have not changed as new observations are "observed" is illogical in my opinion.

The obvious reason is that the "new observation" may be an anomaly to the history and should be classified/identified as so thus the new forecast will go forward without being blindsided by the identified anomaly. Statistical modelling is all about challenging the data and questioning anomalies so that the forecast represents the signal and not aberrant values. Of course if the preferred/used procedure e.g. auto.arima ignores anomalies and assumes that all values are reported without anomalies the question is really moot.