Cross Validation for Time Series Classification (Not Forecasting!)

Question

Is it possible to use regular k-fold cross validation where the folds contain entire time series in time series classification? I'm asking because most sources discussing cross validation with time series say a specific rolling window approach should be used.

I'm doing a time series classification project where I have a large number of multivariate time series of equal length with the same dimensions. I understand that one should not arbitrarily split the time series themselves in the time dimension, and that a sliding window should be used in such a case (where the folds are supersets of preceding folds). However, I do not understand why the folds cannot contain entire time series (e.g training on Time Series 0-100, testing on 101-200, then training on time series 101-200 and testing on 0-100).

I also noticed that in sktime (a python package for time series analysis) the rolling window approach to cross validation splitting was only implemented in their forecasting module, and not in their classification module. Therefore I'm wondering if the sliding window approach to cross validation is only relevant for forecasting tasks where you perhaps only have one time series (and therefore must split in the time dimension) and not for time series classification (where you have multiple entire time series that can be used in distinct folds).

Answer 1

There is no problem with using K-fold cross-validation using the entire time series in time series classification. It is commonly used with algorithms such as NN-DTW (Nearest Neighbour Dynamic Time Warping) to select the size of the warping window. The sktime package you referred to uses this type of cross-validation to select hyper-parameters for some of its classification algorithms - for example, Hive-Cote Lines, Taylor, and Bagnall: HIVE-COTE: The Hierarchical Vote Collective of Transformation-Based Ensembles for Time Series Classification.

Answer 2

I understand that your problem is that your dataset is a time series but that you are not interested in making forecast (i.e. making prediction of future state from past state). In this case, you should avoid random shuffling in your cross-validation because it may lead to overfitting. Indeed, you would have points in your testing set that are surrounded by points in your training set. Therefore, the performance of your model would be unrealistically high.

The scikit-learn user guide has an interesting note about this : https://scikit-learn.org/stable/modules/cross_validation.html#a-note-on-shuffling