0
$\begingroup$

So it turns out I have misunderstood what k-fold CV actually does. I had originally thought that (e.g.) 5-fold CV splits the whole dataset into 5 subsets, then on each iteration the model is trained with 4 and tested with the held-out subset.

But various documentation (https://scikit-learn.org/stable/modules/cross_validation.html) says an additional subset is left out of this procedure, and that the model is trained, validated, and tested? So why is testing the model using the held-out subsets alone not sufficient?

And how does this overcome the problems associated with just using train_test_split, given that you still have to define a subset of the data for testing the final model?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The main reason is that you usually want to make modeling decisions based on the cross-validation results (e.g. what model to choose, what hyperparameter settings for models etc.) and/or use the out-of-fold results to do stacking. Doing either of the two things has the potential to result in some degree of overfitting to the left-out-validation parts of each fold. In contrast a test set that you only ever use once avoids that.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Excellent, thanks for your reply - so how does k-fold CV overcome the bias involved with using train_test_split, if you have to define a separate testing subset anyway? $\endgroup$
    – T.Murray
    Nov 18, 2022 at 14:41
  • 1
    $\begingroup$ k-fold CV on the training set gets used to make modeling decisions and we accept that they way we do it may lead to use getting a biased estimate of model performance. A split into training (within which you'd do CV) and test set gets you an unbiased estimate of performance (if you only assess it once) on the test set. Of course, if you reserve only, say, 20% of the data as a test set, it may turn out that the performance estimate form the CV is still in a sense better, because it uses 80% of the data (i.e. is lower variance). This is an example of a bias-variance trade-off. $\endgroup$
    – Björn
    Nov 18, 2022 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.