How to apply cross-validation for time series analysis using a regression-based approach?

Question

I'd like to know how to use cross-validation for time series analysis using a regression-based approach without incurring in under- or over-fitting. In particular, assume we have an input time series $u$ and an output time series $y$, both of size $n$. Also assume that we want to obtain a regression model of the form ${ y }_{ t }={ \alpha }_{ 1 }{ y }_{ t-1 }+{ \alpha }_{ 2 }{ y }_{ t-2 }+{ \alpha }_{ 3 }{ u }_{ t-5 }+{ e }_{ t }$. Therefore we build a system of equations of the form $Ax=b$, where:

$A=\begin{bmatrix} { y }_{ 5 } & { y }_{ 4 } & { u }_{ 1 } \\ { y }_{ 6 } & { y }_{ 5 } & { u }_{ 2 } \\ \vdots & \vdots & \vdots \\ { y }_{ n-1 } & { y }_{ n-2 } & { u }_{ n-5 } \end{bmatrix}$ $b=\begin{bmatrix} { y }_{ 6 } \\ { y }_{ 7 } \\ \vdots \\ { y }_{ n } \end{bmatrix}$ $x=\begin{bmatrix} { \alpha }_{ 1 } \\ { \alpha }_{ 2 } \\ { \alpha }_{ 3 } \end{bmatrix}$

How should I split the data into training and testing sets for cross-validation given the high dependency within the data?

Answer 1

I don't believe there is any principled answer to this without considering the problem domain. As you pointed out, there may be strong temporal dependencies within the dataset, and there's no obvious way to avoid these. You could try to select training and testing portions which are temporally separated from one another by some buffer zone, but this will require making assumptions about the timescale of the temporal dependencies.

As discussed by Rob Hyndman, one way of measuring model accuracy is one-step-ahead prediction, e.g. fit to timepoints 1-T, then predict $y_{T+1}$ using your model. This doesn't necessarily tell you about generalization performance (e.g. on totally new datasets) but is at least a good within-dataset measure of performance.

In a real application, you will typically have multiple timecourses which are (hopefully) independent, allowing you to cross-validate over different splits.