1
$\begingroup$

While measuring forecast accuracy using the rolling period approach:

  • Should we re-calculate the model in each step. Ex: execute auto.arima() for each new window and calculate the error

OR

  • Calculate the model on the initial window, use the parameters from this model to calculate accuracy at each step.

Rob Hyndman discusses both the approaches in his blog here - https://robjhyndman.com/hyndsight/rolling-forecasts/

However, I am not sure which is a preferred approach.

$\endgroup$
1
$\begingroup$

The purpose of pseudo-out-of-sample testing is to replicate real-life conditions in a controlled experiment. So, once you are done testing your models, what will you do when you actually have to produce real forecasts? Will you select a new model each time a new observation comes along? Or choose it once and then never touch it again? Or maybe at fixed intervals, or when a forecast comes out that "looks bad"?

Your pseudo-out-of-sample testing should replicate exactly the strategy that you will actually use once it's time to forecast for real. If you're going to change the model at every new observation in reality, then the errors you get in testing with a fixed model won't be representative, and vice-versa.

The question of which refitting/model selection frequency is best has no general answer and depends on your data. Refitting too often could lead to overfitting, and not often enough could miss real changes in the process. You would also have to consider the time it takes to select a model and the costs involved each time.

$\endgroup$
  • $\begingroup$ Refitting does not in any way necessarily lead to overfitting unless your software fails to do necessity testing (auto.arima for example) Remodelling premises that there that there might be a real change in the process which needs to be correctly identified and implemented . In general I agree with your over-arching comment that you should duplicate the real process that you employ. If you are not re-modelling after new data is observed with new data then don't use that strategy to evaluate methodology. $\endgroup$ – IrishStat Feb 21 '18 at 15:45
  • $\begingroup$ I said "could lead to", not "necessarily leads to", and I also don't believe that any software is immune to overfitting in every situation. $\endgroup$ – Chris Haug Feb 21 '18 at 16:56
  • $\begingroup$ no piece of software is immune to overfitting . The salient point is there built-in strategies/heuristics that minimize the occurrence of overfitting .You might try a free download of AUTOBOX whwn you get a chance. We are always looking for counter-examples to further strengthen strategies. $\endgroup$ – IrishStat Feb 21 '18 at 20:23
0
$\begingroup$

At any one origin ALL the known historical data should be used to form the best model and a set of parameters and a forecast. To assume that neither the best model has not changed or prior estimates of the best parameters have not changed as new observations are "observed" is illogical in my opinion.

The obvious reason is that the "new observation" may be an anomaly to the history and should be classified/identified as so thus the new forecast will go forward without being blindsided by the identified anomaly. Statistical modelling is all about challenging the data and questioning anomalies so that the forecast represents the signal and not aberrant values. Of course if the preferred/used procedure e.g. auto.arima ignores anomalies and assumes that all values are reported without anomalies the question is really moot.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.