How many times should we repeat a K-fold CV?

Question

I came across this thread looking at the differences between bootstrapping and cross validation - great answer and references by the way. What I am wondering now is, if I was to perform repeated 10-fold CV say to calculate a classifier's accuracy, how many times n should I repeat it?

Does n depend on the number of folds? On the sample size? Is there any rule for this?

(In my case, I have samples as big as 5000, and if I choose anything larger than n = 20 my computer takes way too long to perform the calculation. )

Answer 1

The influencing factor is how stable your model - or, more precisely: the predictions of the surrogates are.

If the models are completely stable, all surrogate models will yield the same prediction for the same test case. In that case, iterations/repetitions are not needed, and they don't yield any improvements.

As you can measure the stability of the predictions, here's what I'd do:

• Set up the whole procedure in a way that saves the results of each cross validation repetition/iteration e.g. to hard disk
• Start with a large number of iterations
• After a few iterations are through, fetch the preliminary results and have a look at the stability/variation in the results for each run.

• Then decide how many further iterations you want to refine the results.

• Of course you may decide to run, say, 5 iterations and then decide on the final number of iterations you want to do.

(Side note: I typically use > ca. 1000 surrogate models, so typical no of repetitions/iterations would be around 100 - 125).

Answer 2

Ask a statistician any question and their answer will be some form of "it depends".

It depends. Apart from the type of model (good point cbeleites!), the number of training set points and the number of predictors? If the model is for classification, a large class imbalance would cause me to increase the number of repetitions. Also, if I am resampling a feature selection procedure, I would bias myself towards more resamples.

For any resampling method used in this context, remember that (unlike classical bootstrapping), you only need enough iterations to get a "precise enough" estimate of the mean of the distribution. That is subjective but any answer will be.

Sticking with classification with two classes for a second, suppose you expect/hope the accuracy of the model to be about 0.80 . Since the resampling process is sampling the accuracy estimate (say p), the standard error would be sqrt[p*(1-p)]/sqrt(B) where B is the number of resamples. For B = 10, the standard error of the accuracy is about 0.13 and with B = 100 it is about 0.04. You might use that formula as a rough guide for this particular case.

Also consider that, in this example, the variance of the accuracy is maximized the closer you get to 0.50 so an accurate model should need less replications since the standard error should be lower than models that are weak learners.

HTH,

Max