1
$\begingroup$

Questions:

  1. Are rolling forecast examples (like the ones below) only useful for evaluating a model's accuracy, or can a rolling forecast be used to train a model?
  2. Are models trained using a rolling forecast generally more accurate?
  3. Can anyone point out an example of a model being trained using a rolling window/rolling forecast technique and forecasted horizons in to the future? By that I mean forecasted horizons beyond the training/testing data used in the rolling forecast.

Examples:
http://robjhyndman.com/hyndsight/tscvexample/
http://robjhyndman.com/hyndsight/rolling-forecasts/

Code:

library("fpp")

h <- 5
train <- window(hsales,end=1989.99)
test <- window(hsales,start=1990)
n <- length(test) - h + 1
fit <- auto.arima(train)
fc <- ts(numeric(n), start=1990+(h-1)/12, freq=12)
for(i in 1:n)
{  
  x <- window(hsales, end=1989.99 + (i-1)/12)
  refit <- Arima(x, model=fit)
  fc[i] <- forecast(refit, h=h)$mean[h]
}
$\endgroup$
1
  • $\begingroup$ This is interesting, as many sources recommend using a rolling window, but there are exceptionelly hard to find any empirical evidence showing that rolling window forecasts are more accurate than forecasts on a fixed window. $\endgroup$
    – Johanna W
    Jun 9, 2023 at 15:28

1 Answer 1

1
$\begingroup$
  1. For a given functional form (e.g. for a given order of ARIMA model), estimating the model using all available data is more efficient than estimating it on a subset of the data. This holds if the data is generated by a process that does not change in time. If, on the other hand, the data generating process itself evolves over time, "old" data may be unrepresentative for a "late" period in the sample, and "new" data may be unrepresentative for an "early" period in the sample. Then discounting or completely dropping early observations may help capture the recent state of the data generating process, which should be useful for forecasting the yet-unobserved data. In other words, rolling windows may come in handy. They may also help assess whether a model estimated on an "early" subsample continues to deliver stable forecasting performance throughout the rest of the sample. If it does not (e.g. the performance worsens with time), it is an indication that the data generating process may be evolving over time.
  2. See 1. for a theoretical argument. I cannot offer empirical evidence, though.
  3. I think this strategy would be more relevant for model selection (e.g. selection of the AR and MA orders in an ARMA model) rather than estimation of a model that has a fixed functional form (e.g. fixed ARMA orders). This is because you would like to use all available data for estimating the model once its functional form has been selected. (Omitting some data is generally inefficient.)
$\endgroup$
4
  • $\begingroup$ Thank you very much for your answer. I'm still trying to understand the steps in the example. What is the purpose of the "refit" step? The model has already been determined and trained by auto.arima in the "fit" step. So are we re-training the model on new data in the "refit" step. If so, what does that accomplish? $\endgroup$
    – ndderwerdo
    Apr 21, 2016 at 3:14
  • $\begingroup$ Hi RIchard, could you please go in to more detail on your response to part 3, that a rolling forecast technique is more relevant to model selection? I'm wondering how you could use a rolling forecast to select an ARIMA model? It seems like in the Hyndman examples the order of the Arima model is already set, or you use auto.arima to fit a model and then evaluate it in the forloop step in the rolling forecast. $\endgroup$
    – ndderwerdo
    Apr 23, 2016 at 22:22
  • $\begingroup$ @Richard Hardy not that I doubt you but are you able to provide a source or link which confirms that "estimating the model using all available data is more efficient than estimating it on a subset of the data"? I am curious because if we assume the data generating function has indeed not changed over time then at some point wouldn't the "excess" data cause the fitting process of the model to become less efficient as there are more data points which do not contribute any "new" information? $\endgroup$
    – Aesir
    Nov 22, 2018 at 15:56
  • 1
    $\begingroup$ @aesir, look at any reasonable estimator's asymptotic distribution and you will find that its variance is decreasing with $n$, the sample size. This holds for OLS, GMM and maximum likelihood, among other estimation techniques. I think this is a rather fundamental principle that more data from the same data generating process is always better, based on the law of large numbers. $\endgroup$ Nov 22, 2018 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.