I would like to use cross validation both to tune the hyperparameters for my supervised learning model, and to perform feature selection. Is it a bad practice to use cross-validation more than once on the same data? Is there any risk of overfitting the cross-validation datasets or anything else that I may be missing?

Thank you


Is it a bad practice to use cross-validation more than once on the same data?

If done properly, no. Nested cross validation would even be one of the recommended techniques for your scenario.

However, while in principle you can nest as deep as you want, the diminishing sample sizes make this impractical. However, which features to use is just a hyperparameter of your modeling process - you can include it with the optimization of the other hyperparameters.

  • $\begingroup$ Don't agree with nested cross validation. You should just cross validate the tuple. $\endgroup$ – seanv507 May 24 '17 at 20:03
  • $\begingroup$ @seanv507: please explain what exactly you mean with "cross validate the tuple". $\endgroup$ – cbeleites supports Monica May 26 '17 at 19:23
  • $\begingroup$ @seanv507: If you refer to the fact you never need more than one level of cross validation for the optimization part (that's what I tried to express in the last sentence), you are of course right. Deep nesting, however, is not bad in the sense of getting you into overfitting: the outer cross validation will be an honest estimate of the generalizatoin performance of your optimized training. $\endgroup$ – cbeleites supports Monica Apr 6 '18 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.