I am looking for a reference (a citable paper) about repeating the cross-validation to decrease the variance of the results.


1 Answer 1


First of all, you may have a misconception: repeated/iterated cross validation will not lead to lower uncertainty on the cross validation results iff the models are stable.

If your models are unstable, evaluating more iterations/repetitions/runs of the cross validation allows you to characterize the instability of the predictions (variance in predictions of different surrogate models that were trained on slightly different/perturbed training data) - and the average over many such surrogate models will have lower variance from this source. Personally, I consider estimates of prediction stability as their own figure of merit (though it is good to be aware of the variance contribution to the cross validation error estimate).

The uncertainty (variance) due the limited number of actual cases tested which does not increase by repeating the cross validation is not affected. Some of our paper could be relevant (but they each contribute only bits and pieces to this topic):

Note that I argue from the perspective of an applied scientist who is interested in estimating the predictive performance of a model built on the data set at hand and at the same time is typically hampered by very small sample sizes (both absolute and relative to the number of variates).
The sources of error [of the CV results] I consequently focus on are quite different e.g. from the perspective of comparing training algorithms and from the perspective of having somewhat more comfortable sample sizes (in particular, absolute numbers of independent cases) at hand.

E.g. Vanwinckelen, G. & Blockeel, H. On estimating model accuracy with repeated cross-validation BeneLearn 2012: Proceedings of the 21st Belgian-Dutch Conference on Machine Learning, 2012, 39-44 argue that repeated cross validation is not worth while, because they find bias more important than the variance reduction. They have about 2 orders of magnitude the sample sizes I typically deal with and do not discuss model stability at all.

  • $\begingroup$ Thanks. The last paper is quite important for what I wanted to do. It shows that although the variance is decreased, the mean does not converge to a number one would get on independent test sample, which is what we are actually trying to estimate. $\endgroup$
    – rep_ho
    Feb 17, 2017 at 12:57
  • $\begingroup$ The pessimistic bias of resampling validation schemes (cross validation, out-of-bootstrap) is well known. Whether that bias or the finite absolute sample size model instability- (sample size relative to model complexity) is the dominating contributor to your validation uncertainty depends a lot on circumstances. $\endgroup$ Feb 18, 2017 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.