Train Test Validation standard split vs Cross Validation

Question

This is a simple question… I am confused with the conceptual difference between a Train | Validation | Test split and K-fold validation.

In K-fold, I understood, We train and validate on everything simultaneously. This is more preferred than having a standard validation set right?

What I mean is, 70 % Training, 15% Validation and 15% Testing

Instead of doing the above method cross validation, Why cant we use K folds and use 85% for both training and validating and remaining 15% as testing data?

Answer 1

You can certainly do inner k-fold CV for model optimization and a single (outer) 15% split for estimating generalization error.

However, as Frank Harrel commented, this is inefficient:

• when doing a single split of your available data, you encounter exactly the same risks for not splitting into independent subsets that you encounter e.g. with k-fold. So no advantage here for the single split.
• Plus, you test for generalization error with a much smaller test sample size and thus your generalization error estimate is subject to more random uncertainty (variance) than it would be with k-fold. More precisely, we expect variance to be almost 7 times as large (1/.15).
  So substantial disadvantage here - unless you have a huge data set at hand so that this improvement is not needed. In that case, you typically wouldn't need the inner/optimization performance estimate to be k-fold, neither.