14
$\begingroup$

After reading one of the "Research tips" of R.J. Hyndman about cross-validation and time series, I came back to an old question of mine that I'll try to formulate here. The idea is that in classification or regression problems, the ordering of the data is not important, and hence k-fold cross-validation can be used. On the other hand, in time series, the ordering of the data is obviously of a great importance.

However, when using a machine learning model to forecast time series, a common strategy is to reshape the series $\{y_1, ..., y_T\}$ into a set of "input-output vectors" which, for a time $t$, have the form $(y_{t-n+1}, ..., y_{t-1}, y_{t}; y_{t+1})$.

Now, once this reshaping has been done, can we consider that the resulting set of "input-output vectors" do not need to be ordered? If we use, for example, a feed-forward neural network with n inputs to "learn" these data, we would arrive at the same results no matter the order in which we show the vectors to the model. And therefore, could we use k-fold cross-validation the standard way, without the need to re-fit the model each time?

$\endgroup$
2
$\begingroup$

The answer to this question is that this will work fine as long as your model order is correctly specified, as then the errors from your model will be independent.

This paper here shows that if a model has poor cross-validation will underestimate how poor it actually is. In all other cases the cross-validation will do a good job, in particular, a better job than the out-of-sample evaluation usually used in the time series context.

$\endgroup$
6
$\begingroup$

Interesting question!

The approach you describe is certainly very widely used by people using standard ML methods that require fixed-length feature vectors of attributes, to analyse time series data.

In the post that you link to, Hyndman points out that there are correlations between the reshaped data vectors (samples). This could be problematic, as k-CV (or other evaluation methods that divide data at random into training and testing sets) assumes that all samples are independent. However, I don't think this concern is relevant for the case of a standard ML methods, that treat attributes separately.

For explanation, let me simplify your notation by assuming $n=3$, so the the first few data vectors (labelled alphabetically) will be: \begin{align} A&: (y_1, y_2, y_3; y_4) \\ B&: (y_2, y_3, y_4; y_5) \\ C&: (y_3, y_4, y_5; y_6) \\ \end{align}

Clearly, A and B have terms such as $y_2$ in common. But, for A, this is the value of its second attribute whereas for B this is the value of its first attribute.

$\endgroup$
  • 1
    $\begingroup$ I agree with you that some ML algorithms may be immune to the problem of highly correlated samples because they treat attributes completely separately. But those algorithms are also not very good for time series work. The ML algorithms that are promising for a time series have to be able to notice that attribute #1 and attribute #2 are actually kinda similar, otherwise they are going to be bad at prediction (the prediction should be roughly similar when you shift time by 1). Those algorithms would also suffer from the issue mentioned by Hyndman. $\endgroup$ – max Jun 14 '16 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.