Repeated Nested Cross validation

Question

I'm aware that nested cross-validation is used for hyperparameter tuning and model selection and that repeated k-fold cross-validation is used to improve the estimated performance of the model.

My question is: can nested cross-validation be repeated?

As described here, nested K-fold cross validation works as follows:

1. Partition the training set into ‘K’ subsets
2. In each iteration, take ‘K minus 1’ subsets for model training, and keep 1 subset (holdout set) for model testing.
3. Further partition the ‘K minus 1’ training set into ‘K’ subsets, and iteratively use the new ‘K minus 1’ subset and the ‘validation set’ for parameter tuning (grid search). The best parameter identified in this step is used to test on the holdout set in step 2.

Can this process be repeated N times, for different partitions of the dataset, just as you would do in repetead cross-validation?

I came across this question which seems related to my question, but unfortunately has no accepted answer.

I also came across this paper where they seem to describe the procedure I'm asking about, though I haven't seen it described elsewhere.

Answer 1

Yes, you can repeat the cross validations (both or any of them, as necessary).

---

Repeated cross validation improves the estimate of generalization error in a specific way: it allows you to easily measure and reduce random uncertainty due to model instability (unstable predictions).

I find it more convenient to think of the nested cross valiation a bit differently:

• Outer validation: aim is to measure generalization error of a fully tuned, useable model. Repeated CV is a good candidate for this.
• Training of a fully tuned model: As usual, this self-tuning training function is a black box to the outer validation. The training procedure can internally use further (inner) cross validations, which of course can be repeated as well.

IMHO the advantages of this point of view are

• It is immediately clear that the details of the validations can be choosen independently for the outer vs. inner validation procedures.
  • You can adapt/organize the inner validation procedure as suitable for the type of model you tune.
• It will automatically lead to a correctly nested setup.
• Coding this way guards against indexing errors that leak outer test data into the tuning part.
• The final model can be trained as usual: by calling the self-tuning training function on the whole data set.

Answer 2

I would correct your first sentence as follows:

• Nested cross-validation is used to estimate the out-of-sample performance of the prediction algorithm (aka generalization error or true prediction error)
• Repeated k-fold cross-validation (instead of using a single K-fold split) is often applied to enchance hyperparameter tuning. Tt can also be applied in nested cross-validation to improve error estimation.