0
$\begingroup$

This is a simple question… I am confused with the conceptual difference between a Train | Validation | Test split and K-fold validation.

In K-fold, I understood, We train and validate on everything simultaneously. This is more preferred than having a standard validation set right?

What I mean is, 70 % Training, 15% Validation and 15% Testing

Instead of doing the above method cross validation, Why cant we use K folds and use 85% for both training and validating and remaining 15% as testing data?

$\endgroup$
2
  • 2
    $\begingroup$ This has been covered at length in other posts. Data splitting is arbitrary and terribly inefficient. Only an enormous sample size can overcome problems caused by data splitting. $\endgroup$ Commented Nov 9, 2021 at 12:52
  • $\begingroup$ Explain what you mean by enormous. Because I’ve seen you write this before and then say a data set needs at least 20,000 records before and 20,000 records is absolutely tiny in today’s world $\endgroup$
    – astel
    Commented Nov 9, 2021 at 18:27

1 Answer 1

2
$\begingroup$

You can certainly do inner k-fold CV for model optimization and a single (outer) 15% split for estimating generalization error.

However, as Frank Harrel commented, this is inefficient:

  • when doing a single split of your available data, you encounter exactly the same risks for not splitting into independent subsets that you encounter e.g. with k-fold. So no advantage here for the single split.
  • Plus, you test for generalization error with a much smaller test sample size and thus your generalization error estimate is subject to more random uncertainty (variance) than it would be with k-fold. More precisely, we expect variance to be almost 7 times as large (1/.15).
    So substantial disadvantage here - unless you have a huge data set at hand so that this improvement is not needed. In that case, you typically wouldn't need the inner/optimization performance estimate to be k-fold, neither.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.