Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Are there any good reasons to prefer a sliding model training window to a growing window in online time series forecasting (or vice versa)? I'm particularly referring to financial time series.

I would intuitively think a sliding window should perform worse -- out of sample-- as it is has more potential for over-fitting specific sample window characteristics, but some of the empirical results I've seen are counter to this.

Also, given that a sliding window is preferred by some, what would your approach be to determine the look-back length (any good reasons to prefer one over another, aside from pure heuristics)?

Although I didn't specify a model, an example might be ARIMA.

EDIT: I should add there there is a related blog post by By. Rob Hyndman, with what he dubbed 'time-series' cross validation. While it does cover the concepts described, it doesn't give much of a formal reason about why one method might be preferential over the other, nor any ideas about an optimal look-back window parameter.

share|improve this question
add comment

2 Answers

up vote 4 down vote accepted

The choice of window length involves a balance between two opposing factors. A shorter window implies a smaller data set on which to perform your estimations. A longer window implies an increase in the chance that the data-generating process has changed over the time period covered by the window, so that the oldest data are no longer representative of the system's current behavior.

Suppose, for example, that you wished to estimate January mean temperature in New York. Due to climate change, data from 40 years ago are no longer representative of current conditions. However, if one uses only data from the past 5 years, your estimate will have a large uncertainty due to natural sampling variability.

Analogously, if you were trying to model the behavior of the Dow Jones Industrial Average, you could pull in data going back over a century. But you may have legitimate reasons to believe that data from the 1920s will not be representative of the process that generates the DJIA values today.

To put it in other terms, shorter windows increase your parameter risk while longer windows increase your model risk. A short data sample increases the chance that your parameter estimates are way off, conditional on your model specification. A longer data sample increases the chance that you are trying to stretch your model to cover more cases than it can accurately represent. A more "local" model may do a better job.

Your selection of window size depends, therefore, on your specific application -- including the potential costs for different kinds of error. If you were certain that the underlying data-generating process was stable, then the more data you have, the better. If not, then maybe not.

I'm afraid I can't offer more insight on how to strike this balance appropriately, without knowing more about the specifics of your application. Perhaps others can offer pointers to particular statistical tests.

What most people do in practice (not necessarily the best practice) is to eyeball it, choosing the longest window for which one can be "reasonably comfortable" that the underlying data-generating process has, during that period, not changed "much". These judgements are based on the analyst's heuristic understanding of the data-generating process.

share|improve this answer
    
Another reason to use a fixed window length: consecutive model results are, arguably, more completely comparable than if the model window is growing. If your forecast error appears to be decreasing over time, you can interpret that as indicating a decline in the volatility of the underlying process: you don't need to try to tease out how much of the decline should instead be attributed to the fact that parameter estimates are being made on the basis of increasingly large data sets. –  Arthur Small Dec 20 '12 at 21:49
    
I completely agree with your observations. But I was looking for a more formal reason to prefer one method over the other (maybe it has to be trial and error or intuition/heuristic based). I can say that most time series texts don't really address this issue much. –  pat Dec 20 '12 at 22:13
    
:Arthur The Chow Test can be used to test the equality/constancy of coefficients over time . Additionally Tsay's test unc.edu/~jbhill/tsay.pdf for constancy of variance can also be employed. We have incorporated both of these features into AUTOBOX , a piece of commercial software that I am involved with. –  IrishStat Dec 21 '12 at 14:15
add comment

I am a little bit confused by your language. Do you mean how do I weight the past to make a prediction e.g. a 12 period moving average ? An ARIMA model determines the number of observations to be used (window ? ) and how to optimally weight the values in the window. One example is to assign a weight of 1 to the last observation and zero elsewhere . Another example might be to equally weight the last k observations. An ARIMA model can be expressed in these terms (Pi Weights).

Your first proposal: THe updating could be as simple as tuning the parameters or tuning paramnters and possibly changing the model. This approach would give you an honest estimate of out-of-sample performance. Your second proposal would be better if either the parameters were changing , or the model was changing or the error variance eas changing as only new data would be used. The problem with that is discarding/ignoring previous values comes at a price with respect to model formulation. Recently I suggested to a client that they take 1200 historical(daily values) and predict 30 days out at some 20 time points in the past and measure the weighted MAPE for a 30 day period. The forecasts at each point would be premised on data up to that point and care given to validating the the parameters/model had not changed up to that point and that the error variance had passed the variance constancy requirement. In this way they could have an empirical distrivution of the expected MAPE from any future origin to use as a benchmark or expected MAPE. In this way unusual mapes vould be identified. Unusual MAPE's can suggest a change in the DGP ( Data Generation Process) or a deficiciency in the Modelling Process. It appears to me that your primary concern is to sense/challenge hoe many points are actually used to form a model. Which by the way has been a primary focus of mine /

share|improve this answer
    
Sorry for any confusion. Imagine we have 100 yr of daily Dow obs. We want to fit ARIMA model(s) and predict 1 day forward. We could take first 90 years of prior data and fit our parameters over that prior window- then, each day forward record the actual and predicted observation, then update our model (growing)-- and repeat till all data was consumed. We would then have 10 years of observations to get empirical out of sample performance results. –  pat Dec 20 '12 at 22:30
    
...Or we could take 1 year sample windows to model the parameters, record the 1 day fwd est and actual observations. Then each period (day) slide the 1 year window fwd and repeat. The results of both methods will obviously differ, but which is a better model to dynamically utilize in an online fashion and/or why. –  pat Dec 20 '12 at 22:30
    
Your first proposal: THe updating could be as simple as tuning the parameters or tuning paramnters and possibly changing the model. This approach would give you an honest estimate of out-of-sample performance. –  IrishStat Dec 21 '12 at 0:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.