8
$\begingroup$

Cross-validation (CV) splits the data into two portions, one for building the model and one for testing it.

A common practice is to repeat CV to get more precise estimates of the model's performance. For example, instead of doing CV only once, it is repeated 100 times with random splits and the mean performance is reported.

Besides increased computation time, what is the disadvantage of this approach?

Does it increase model bias or variance?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Can you elaborate on repeat CV? $\endgroup$
    – gunes
    Commented Feb 4, 2020 at 16:59

2 Answers 2

Reset to default
4
$\begingroup$

There is no disadvantage in doing repeated CV in comparison with a single CV fold. If anything, repeated CV should decrease the variance of our estimate.

An excellent and highly-cited overview on cross-validation procedures can be found in Arlot & Celisse (2010) A survey of cross-validation procedures for model selection. The paper is admittedly a bit long (it is a survey after all) but even reading the last Section on "Conclusion: which cross-validation method for which problem?" is very enlightening. It discusses how no single CV procedure is universally better but we should focus on the particular setting (e.g. variable selection vs. choosing the best among two learning procedures).

Improve this answer
$\endgroup$
4
  • $\begingroup$ But what about the overlap of training/testing data when repeating cross-validation? Doesn't this have some negative implications? $\endgroup$
    – Funkwecker
    Commented Feb 6, 2020 at 12:57
  • $\begingroup$ It will have no serious implications because the performance from each single fold is evaluated separately and the folds are created independently within each repeat. Some very minor correlation might be seen between the repeats but after all we are sampling the same "population", effectively it is the same source of correlation we would expect when bootstrapping (but even smaller for $K>3$). $\endgroup$
    – usεr11852
    Commented Feb 6, 2020 at 17:17
  • $\begingroup$ Thanks, following your argument 10 times 10-fold cross-validation is as good as 10 times cross-validation with a 90/10 split. Thus, when repeating CV, using folds is not necessary, right? $\endgroup$
    – Funkwecker
    Commented Feb 7, 2020 at 11:07
  • $\begingroup$ If we do not use folds we are just bootstrapping (albeit with a slightly smaller sample). I was referring to potential source of spurious correlations (the "negative implications" in your original comment). $\endgroup$
    – usεr11852
    Commented Feb 7, 2020 at 11:19
1
$\begingroup$

With regards to the question of disadvantage, I think that needs refining.

Disadvantage compared to k-fold CV? For large samples, it's computational time (as you noted). For small samples, there is no apparent disadvantage for repeated k-fold CV.

Disadvantage compared to bootstrapping? For small samples, bootstrapping can be better at choosing between models because it will pick up issues with important variables being dropped. At larger samples, bootstrapping can be problematic because of overfitting. This article looks at bias and variance between bootstrapping and repeated 10-fold CV, with a note that they are surprised by the low variance of the repeated 10-fold CV resampling compared to bootstrapping.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.