Colleagues of mine were tasked with building a forecasting model. They intended on doing a train/test split for cross validation. I know that when training models for forecasting, a "walk forward" validation is more often used. Why is that?

  • $\begingroup$ Because usually your model for the data $Y_t$ at time $t$ will depend on its past values $Y_{t-1}, Y_{t-2}, \dots, Y_{t-p}$. So when you validate your prediction $\widehat{Y}_t$, you need $Y_{t-1}, Y_{t-2}, \dots, Y_{t-p}$ to do so. In contrast, in the cross-sectional setting, $Y_t$ will not depend on its own past and there is also no natural order (like the time ordering of a time series) in the data. $\endgroup$ – Jeremias K Dec 9 '17 at 19:07

While I think the comment to your post covers the 'why', what you mention is generally called a 'rolling forecast origin', which is alright for one-step forecasts.

Instead, try using a similar concept (if you have enough data) where you define a test set of length h, remove h of your most recent data points, run your models on the remaining data, record accuracy of the forecasts versus the withheld data. Then iterate the process until you run out of enough training data to build models (perhaps 3 seasonal cycles). Average the out-of-sample error results and you'll have a good idea of which should perform best. This is the time-series equivalent of CV.

  • $\begingroup$ Could you clarify if by remaining data you mean "the data older than h items" or "all past and future data except h items"? $\endgroup$ – elemolotiv Dec 27 '18 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.