1
$\begingroup$

In the paper "Evaluating time series forecasting models: an empirical study on performance estimation methods" by Cerqueira et al (2020), they mention k-fold cross-validation. Which they differentiate from blocked cross-validation in subsection 2.3.1.

I thought I understood the difference but if I then check sklearn's KFold, I am not sure anymore if this is the k-fold cross validation or the blocked cross-validation. Is the k-Fold from sklearn equivalent to the blocked cross validation mentioned in the paper?

And which method of sklearn is equivalent then to the k-fold mentioned in the paper? Is it Shuffle & Split?

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, the default k-fold splitter in sklearn is the same as this 'blocked' cross validation. Setting shuffle=True will make it like the k-fold described in the paper.

From page 2001 of the paper:

The typical approach when using K-fold cross-validation is to randomly shuffle the data and split it in K equally-sized folds or blocks.

Then, on the next page:

The Blocked Cross-Validation (Snijders 1988) (CV-Bl) procedure is similar to the standard form described above. The difference is that there is no initial random shuffling of observations.

So the difference is the shuffling. We must not shuffle time-series data because the records are not independent from each other. As the authors say:

[k-fold[ cross-validation assumes the observations to be i.i.d.

According to the docs, sklearn.model_selection.KFold does not shuffle by default. As long as it is not shuffling, you could use it for time-series, and it is the same as the 'blocked cross-validation' mentioned in the paper. If you set shuffle=True then it would be the same as the k-fold splitting the authors describe. (ShuffleSplit is a bit different because each fold is random, so records can appear in multiple folds.)

However, a better option for time-series data is probably sklearn.model_selection.TimeSeriesSplit, because it accumulates the training data over time. This is what the authors call 'prequential' sampling, as illustrated in their Figure 3. You can optionally introduce a gap between blocks, which is what the paper authors call 'hv-Blocked Cross-Validation'.

I recommend reading the entire section of the sklearn User Guide devoted to cross validation, it explains everything pretty well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.