9
$\begingroup$

For hyperparameter tuning (random search/ grid search/ bayesian optimization), there are many trials performed for each set of hyperparameters. To evaluate how good a set of hyperparameter is, we can use k fold cross validation which splits the training data into k folds.

Previously, I used to split the training data into k fold and used the same fold splits for all my hyperparameter trials. However, after trying out sklearn Pipelines, it seems that using a pipeline with RandomsearchCV results in a different k fold split for each hyperparameter trial.

I am wondering if it matters at all if we used the same k fold split for all trials or if it is important that we randomized the split for each trial?

$\endgroup$
1

4 Answers 4

6
$\begingroup$

It'd actually be better to use the same folds while comparing different models, as you've done initially. If you input the pipeline object into the randomCV object, it should use the same folds. But, if you do the other way around, each run will change the folds as you said. Even in that case, you can fix the folds by fixing the cv argument in the pipeline object.

$\endgroup$
6
  • $\begingroup$ Oh true, I remember seeing code examples that pass the CV object. One question though, if I pass a cv argument for example, cv=5, it will use the same 5 fold splits. But if my pipeline has a preprocessing step before the model, will the preprocessing part of the pipeline be "re-run" for every trial? If so, isn't this quite wasteful... $\endgroup$ Jan 17, 2021 at 15:18
  • $\begingroup$ Are you inputting your pipeline into RandomCV object or is RandomCV object a step in your pipeline? The first one is a better design. $\endgroup$
    – gunes
    Jan 17, 2021 at 15:41
  • $\begingroup$ @LimKaizhuo you can use the memory option of Pipeline to cache the preprocessing steps on each fold. $\endgroup$ Jan 17, 2021 at 16:59
  • $\begingroup$ @gunes It is the first, e.g. regr = make_pipeline(preprocess, regr) followed by RandomsearchCV(regr, params) $\endgroup$ Jan 17, 2021 at 17:17
  • $\begingroup$ Yes, it'd be repeated but you can follow @BenReiniger's advice on that. $\endgroup$
    – gunes
    Jan 17, 2021 at 17:22
5
$\begingroup$

Using the same or different splits amounts to using a different experimental design for your optimization:

Using the same splits means that you are setting up the comparisons for your optimization in a paired fashion. Paired tests/comparisons typically have higher statistical power which you may use for your optimization decisions.

$\endgroup$
2
$\begingroup$

Keeping same fold for hyperparameter tunning is better idea. If you have random data in each iteration, then you will not be able to understand whether variance in the model is coming due to random data or different hyperparameter. So, to eliminate variance in the model due to randomness in the data, we generally use static folds which can be created before grid or random search starts.

$\endgroup$
1
$\begingroup$

Let me first rephrase the question to make it a little more precise:

"I am wondering if it matters at all if we used the same k fold split for all trials or if it is important that we randomized the split for each trial?"

Assume you perform hyperparameter tuning using fixed folds, and random folds. The two tunings will select, in general, different models as the best. The split method matters if those two models have significantly different performance. Conversely, if the difference in performance is negligible, the choice of fixed or random folds does not matter, because they both select equally good models. I'll set aside for the moment on how you decide if the two selected models are different (not trivial, but it's a separate topic).

To my knowledge there is virtually no literature published on your question. I have used both methods, and have not noticed difference in performance, but have not explored the question systematically. But if the choice of random vs. fixed folds had a significant effect, there would have been published reports about it. My answer is, therefore, in practical sense it doesn't make a difference which method you use

To be sure, cross-validation can produce heavily biased performance estimates for small sample sizes, but neither fixed nor random CV can solve the problem in such datasets. It can be alleviated, to a degree, using repeated CV and nested CV:

https://jcheminf.biomedcentral.com/articles/10.1186/1758-2946-6-10

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.