For hyperparameter tuning with cross validation, is it okay for the fold splits to be same for every hyperparameter trial?

Question

For hyperparameter tuning (random search/ grid search/ bayesian optimization), there are many trials performed for each set of hyperparameters. To evaluate how good a set of hyperparameter is, we can use k fold cross validation which splits the training data into k folds.

Previously, I used to split the training data into k fold and used the same fold splits for all my hyperparameter trials. However, after trying out sklearn Pipelines, it seems that using a pipeline with RandomsearchCV results in a different k fold split for each hyperparameter trial.

I am wondering if it matters at all if we used the same k fold split for all trials or if it is important that we randomized the split for each trial?

Answer 1

It'd actually be better to use the same folds while comparing different models, as you've done initially. If you input the pipeline object into the randomCV object, it should use the same folds. But, if you do the other way around, each run will change the folds as you said. Even in that case, you can fix the folds by fixing the cv argument in the pipeline object.

Answer 2

Using the same or different splits amounts to using a different experimental design for your optimization:

Using the same splits means that you are setting up the comparisons for your optimization in a paired fashion. Paired tests/comparisons typically have higher statistical power which you may use for your optimization decisions.

Answer 3

Keeping same fold for hyperparameter tunning is better idea. If you have random data in each iteration, then you will not be able to understand whether variance in the model is coming due to random data or different hyperparameter. So, to eliminate variance in the model due to randomness in the data, we generally use static folds which can be created before grid or random search starts.

Answer 4

Let me first rephrase the question to make it a little more precise:

"I am wondering if it matters at all if we used the same k fold split for all trials or if it is important that we randomized the split for each trial?"

Assume you perform hyperparameter tuning using fixed folds, and random folds. The two tunings will select, in general, different models as the best. The split method matters if those two models have significantly different performance. Conversely, if the difference in performance is negligible, the choice of fixed or random folds does not matter, because they both select equally good models. I'll set aside for the moment on how you decide if the two selected models are different (not trivial, but it's a separate topic).

To my knowledge there is virtually no literature published on your question. I have used both methods, and have not noticed difference in performance, but have not explored the question systematically. But if the choice of random vs. fixed folds had a significant effect, there would have been published reports about it. My answer is, therefore, in practical sense it doesn't make a difference which method you use

To be sure, cross-validation can produce heavily biased performance estimates for small sample sizes, but neither fixed nor random CV can solve the problem in such datasets. It can be alleviated, to a degree, using repeated CV and nested CV:

https://jcheminf.biomedcentral.com/articles/10.1186/1758-2946-6-10