Time series forecasting lookback windows -- sliding or growing?

Question

Are there any good reasons to prefer a sliding model training window to a growing window in online time series forecasting (or vice versa)? I'm particularly referring to financial time series.

I would intuitively think a sliding window should perform worse -- out of sample-- as it is has more potential for over-fitting specific sample window characteristics, but some of the empirical results I've seen are counter to this.

Also, given that a sliding window is preferred by some, what would your approach be to determine the look-back length (any good reasons to prefer one over another, aside from pure heuristics)?

Although I didn't specify a model, an example might be ARIMA.

EDIT: I should add there there is a related blog post by By. Rob Hyndman, with what he dubbed 'time-series' cross validation. While it does cover the concepts described, it doesn't give much of a formal reason about why one method might be preferential over the other, nor any ideas about an optimal look-back window parameter.

Answer 1

The choice of window length involves a balance between two opposing factors. A shorter window implies a smaller data set on which to perform your estimations. A longer window implies an increase in the chance that the data-generating process has changed over the time period covered by the window, so that the oldest data are no longer representative of the system's current behavior.

Suppose, for example, that you wished to estimate January mean temperature in New York. Due to climate change, data from 40 years ago are no longer representative of current conditions. However, if one uses only data from the past 5 years, your estimate will have a large uncertainty due to natural sampling variability.

Analogously, if you were trying to model the behavior of the Dow Jones Industrial Average, you could pull in data going back over a century. But you may have legitimate reasons to believe that data from the 1920s will not be representative of the process that generates the DJIA values today.

To put it in other terms, shorter windows increase your parameter risk while longer windows increase your model risk. A short data sample increases the chance that your parameter estimates are way off, conditional on your model specification. A longer data sample increases the chance that you are trying to stretch your model to cover more cases than it can accurately represent. A more "local" model may do a better job.

Your selection of window size depends, therefore, on your specific application -- including the potential costs for different kinds of error. If you were certain that the underlying data-generating process was stable, then the more data you have, the better. If not, then maybe not.

I'm afraid I can't offer more insight on how to strike this balance appropriately, without knowing more about the specifics of your application. Perhaps others can offer pointers to particular statistical tests.

What most people do in practice (not necessarily the best practice) is to eyeball it, choosing the longest window for which one can be "reasonably comfortable" that the underlying data-generating process has, during that period, not changed "much". These judgements are based on the analyst's heuristic understanding of the data-generating process.

Answer 2

I am a little bit confused by your language. Do you mean how do I weight the past to make a prediction e.g. a 12 period moving average ? An ARIMA model determines the number of observations to be used (window ? ) and how to optimally weight the values in the window. One example is to assign a weight of 1 to the last observation and zero elsewhere . Another example might be to equally weight the last k observations. An ARIMA model can be expressed in these terms (Pi Weights).

Your first proposal: THe updating could be as simple as tuning the parameters or tuning paramnters and possibly changing the model. This approach would give you an honest estimate of out-of-sample performance. Your second proposal would be better if either the parameters were changing , or the model was changing or the error variance eas changing as only new data would be used. The problem with that is discarding/ignoring previous values comes at a price with respect to model formulation. Recently I suggested to a client that they take 1200 historical(daily values) and predict 30 days out at some 20 time points in the past and measure the weighted MAPE for a 30 day period. The forecasts at each point would be premised on data up to that point and care given to validating the the parameters/model had not changed up to that point and that the error variance had passed the variance constancy requirement. In this way they could have an empirical distrivution of the expected MAPE from any future origin to use as a benchmark or expected MAPE. In this way unusual mapes vould be identified. Unusual MAPE's can suggest a change in the DGP ( Data Generation Process) or a deficiciency in the Modelling Process. It appears to me that your primary concern is to sense/challenge hoe many points are actually used to form a model. Which by the way has been a primary focus of mine /