With a high sample:predictor (n:p) ratio, as opposed to nested CV, why not just go with the split sample approach in which CV is done on training data (e.g., 80%) for model selection and estimation of the generalization error is done on the held out test data (e.g., 20%). The optimization bias at this point should be eliminated since model selection is separated from performance evaluation, so what is the benefit?

  • 1
    $\begingroup$ Evaluating on just one fold of test data rather than five folds is a weaker estimation of generalization error. $\endgroup$ – John Smith Nov 1 '19 at 16:57
  • $\begingroup$ I am not asking about evaluating the generalization error on one fold in CV over the full dataset, but why should we do nested CV over split-sample validation with a large n:p ratio $\endgroup$ – pauluccd927 Nov 1 '19 at 17:13
  • 1
    $\begingroup$ If I understand your question correctly, the split-sample validation you propose corresponds to one "fold" in the outer CV of the nested CV. You don't know if your held out test data is representative, so it's a weaker estimate of generalization error than generating that hold out test data in five folds (as the "outer CV" of the nested CV). $\endgroup$ – John Smith Nov 1 '19 at 19:08
  • $\begingroup$ The split sample validation approach Im referring to is the 'common textbook approach' of taking a full dataset and holding out a single test set (20%), doing model selection via CV on a training set (80%) and evaluating model performance on that single held out test set. My question is getting at the benefit of doing nested CV vs this approach when n is large. But I believe what you are getting at is a similar point @Monica is getting at which is that in essence, your generalization error will be based on more test cases than a single held out test set or even a single fold of the outer loop. $\endgroup$ – pauluccd927 Nov 1 '19 at 20:04

The benefit is that you get a better estimate of the model performance as

  • it is based on absolutely more tested cases (the n : p ratio isn't relevant here, n is relevant)
  • you can check the stability of your modeling approach: if the 10 outer folds arrive essentially at equal models, then your training is stable. If these models vary wildly (not all differences are harmful, though. See also other question), you may need to go back and adapt your hyperparameter (incl. model selection) heuristic to stabilize your whole training procedure.
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.