3
$\begingroup$

I'm aware that nested cross-validation is used for hyperparameter tuning and model selection and that repeated k-fold cross-validation is used to improve the estimated performance of the model.

My question is: can nested cross-validation be repeated?

As described here, nested K-fold cross validation works as follows:

1. Partition the training set into ‘K’ subsets
2. In each iteration, take ‘K minus 1’ subsets for model training, and keep 1 subset (holdout set) for model testing.
3. Further partition the ‘K minus 1’ training set into ‘K’ subsets, and iteratively use the new ‘K minus 1’ subset and the ‘validation set’ for parameter tuning (grid search). The best parameter identified in this step is used to test on the holdout set in step 2.

Can this process be repeated N times, for different partitions of the dataset, just as you would do in repetead cross-validation?

I came across this question which seems related to my question, but unfortunately has no accepted answer.

I also came across this paper where they seem to describe the procedure I'm asking about, though I haven't seen it described elsewhere.

$\endgroup$
1
$\begingroup$

Yes, you can repeat the cross validations (both or any of them, as necessary).


Repeated cross validation improves the estimate of generalization error in a specific way: it allows you to easily measure and reduce random uncertainty due to model instability (unstable predictions).

I find it more convenient to think of the nested cross valiation a bit differently:

  • Outer validation: aim is to measure generalization error of a fully tuned, useable model. Repeated CV is a good candidate for this.
  • Training of a fully tuned model: As usual, this self-tuning training function is a black box to the outer validation. The training procedure can internally use further (inner) cross validations, which of course can be repeated as well.

IMHO the advantages of this point of view are

  • It is immediately clear that the details of the validations can be choosen independently for the outer vs. inner validation procedures.
    • You can adapt/organize the inner validation procedure as suitable for the type of model you tune.
  • It will automatically lead to a correctly nested setup.
  • Coding this way guards against indexing errors that leak outer test data into the tuning part.
  • The final model can be trained as usual: by calling the self-tuning training function on the whole data set.
$\endgroup$
2
  • $\begingroup$ Thank you for the explanation. Do you know what is the limit/recommended number of times that I can repeat nested cross-validation? $\endgroup$ – asere Sep 22 '20 at 11:37
  • 1
    $\begingroup$ The sensible number of repetitions depends on the instability of the models - both absolutely and relative to the uncertainty you have because of a finite sample size. Here's a practical heuristic I use: start with a small number of repetition (say, 5). If things turn out to be unstable, do more repetitions so that either the contribution to total uncertainty becomes acceptable. Observed instability may be a tuning criterion as well. If you tune for stability, you'll probably need only a few repetitions in the outer CV. $\endgroup$ – cbeleites unhappy with SX Sep 22 '20 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.