1
$\begingroup$

Is it possible to use regular k-fold cross validation where the folds contain entire time series in time series classification? I'm asking because most sources discussing cross validation with time series say a specific rolling window approach should be used.

I'm doing a time series classification project where I have a large number of multivariate time series of equal length with the same dimensions. I understand that one should not arbitrarily split the time series themselves in the time dimension, and that a sliding window should be used in such a case (where the folds are supersets of preceding folds). However, I do not understand why the folds cannot contain entire time series (e.g training on Time Series 0-100, testing on 101-200, then training on time series 101-200 and testing on 0-100).

I also noticed that in sktime (a python package for time series analysis) the rolling window approach to cross validation splitting was only implemented in their forecasting module, and not in their classification module. Therefore I'm wondering if the sliding window approach to cross validation is only relevant for forecasting tasks where you perhaps only have one time series (and therefore must split in the time dimension) and not for time series classification (where you have multiple entire time series that can be used in distinct folds).

Improve this question
$\endgroup$
4
  • $\begingroup$ I don't see the issue with using entire time series - the issue is that the blocks need to be statistically independent. Maybe this paper is helpful: Roberts, David R., et al. "Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure." Ecography 40.8 (2017): 913-929. $\endgroup$
    – Florian Hartig
    May 27, 2022 at 10:50
  • $\begingroup$ What do you mean by classification? What exactly are you trying to do? $\endgroup$
    – Richard Hardy
    May 27, 2022 at 11:05
  • $\begingroup$ This type of classification: developersbay.se/time-series-classification-an-overview $\endgroup$
    – perceptronEnthusiast420
    May 27, 2022 at 11:08
  • 1
    $\begingroup$ Direct application of cross-validation on temporal data is not possible due to serial correlation and absurdity of using future data to predict past. Though there are proposal of block sampling, see Stationary Bootstrap. However, we have also proposed a meta algorithm so called rCV at a fixed window and that each fold is tested on the out-of-sample portion. Essentially, we could generate folds by removing data points at random and reconstruct them using interpolation or similar using secondary model. $\endgroup$
    – patagonicus
    May 28, 2022 at 22:15

2 Answers 2

Reset to default
0
$\begingroup$

There is no problem with using K-fold cross-validation using the entire time series in time series classification. It is commonly used with algorithms such as NN-DTW (Nearest Neighbour Dynamic Time Warping) to select the size of the warping window. The sktime package you referred to uses this type of cross-validation to select hyper-parameters for some of its classification algorithms - for example, Hive-Cote Lines, Taylor, and Bagnall: HIVE-COTE: The Hierarchical Vote Collective of Transformation-Based Ensembles for Time Series Classification.

Improve this answer
$\endgroup$
1
$\begingroup$

I understand that your problem is that your dataset is a time series but that you are not interested in making forecast (i.e. making prediction of future state from past state). In this case, you should avoid random shuffling in your cross-validation because it may lead to overfitting. Indeed, you would have points in your testing set that are surrounded by points in your training set. Therefore, the performance of your model would be unrealistically high.

The scikit-learn user guide has an interesting note about this : https://scikit-learn.org/stable/modules/cross_validation.html#a-note-on-shuffling

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.