Ask a statistician any question and their answer will be some form of "it depends".
It depends. Apart from the type of model (good point cbeleites!), the number of training set points and the number of predictors? If the model is for classification, a large class imbalance would cause me to increase the number of repetitions. Also, if I am resampling a feature selection procedure, I would bias myself towards more resamples.
For any resampling method used in this context, remember that (unlike classical bootstrapping), you only need enough iterations to get a "precise enough" estimate of the mean of the distribution. That is subjective but any answer will be.
Sticking with classification with two classes for a second, suppose you expect/hope the accuracy of the model to be about 0.80 . Since the resampling process is sampling the accuracy estimate (say
p), the standard error would be
B is the number of resamples. For
B = 10, the standard error of the accuracy is about 0.13 and with
B = 100 it is about 0.04. You might use that formula as a rough guide for this particular case.
Also consider that, in this example, the variance of the accuracy is maximized the closer you get to 0.50 so an accurate model should need less replications since the standard error should be lower than models that are weak learners.