Bootstrap vs Crossvalidation for assessing the predictive performance of a machine learning model

Question

I'm trying to decide between bootstrap and cross-validation for thoroughly evaluating predictive model performance.

Bootstrap:

• Samples data with replacement, creating B > 1000 (or more) diverse training sets, where each observation is treated idependently due to sampling with replacement! This means that the empirical distribution function is used as a drop-in replacement for the real world distribution from which the samples are drawn.
• Dynamically Use out-of-bootstrap (out-of-bag) (OOB) observations as holdout sets: This eliminates the need for a separate holdout set and can provide a more reliable estimate of predictive out-of-sample performance.
• Potentially captures more real-world variance (by sampling with replacement, i.e. making use of the emprical distribution function)

If I had to evaluate two different models, I would

1. produce B=1000 ... bootstrap resamples of the training dataset
2. evaluate the two models on the B different out-of-bag test sample sets
3. compute paired model performance differences (performance model 1 - performance model 2, where the difference is taken for an identical OOB test set), pairing would be used for https://en.wikipedia.org/wiki/Variance_reduction
4. make a histogram of the paired model performance differences
5. if the density of model performances is visibly shifted away from 0, then one of the models outperforms the other one (from this one could also device some test for 95% certainty of improvement ...)

Cross-Validation:

• Splits data into folds, using different folds for training and testing. Can be more efficient, especially with limited data. Might lead to "correlated" training sets, especially in LOOCV (the training sets almost do not change, except for one observation)

For comparing two models, I would do the same as in the bootstrap case (just without having out-of-bag test samples, but having maximally N=number of observations different hold-out sets in the case of LOOCV).

Question:

Given that bootstrap can introduce more variability in training sets (by resampling with replacement) and leverage OOB data for validation, wouldn't it provide a more realistic estimate of out-of-sample performance compared to cross-validation (which essentially is resampling without replacement)? Is especially the potential "correlation" between cross-validation training sets a significant concern?

PS: I tried to search stackoverflow and came up with similar questions, but they do not discuss the potential correlation problem of CV training sets:

• Bootstrap or jack-knife for crossvalidation of predictive model? (no answer)
• Understanding bootstrapping for validation and model selection
• Differences between cross validation and bootstrapping to estimate the prediction error (no definite answer which is more reliable)
• Cross-validation or bootstrapping to evaluate classification performance? (this question seems to be on the spot, but the accepted answer mainly deals with only having one test-train split (a degenerate form of CV, which clearly suffers from not having multiple test-train splits like in k-fold CV), Frank Harell has an answer https://stats.stackexchange.com/a/71189/298651 which links https://hbiostat.org/doc/simval.html having one conclusion "Ordinary bootstrap is better than or equal to all cross-validation strategies tried", but this is not clearly discussed in his answer ...)
• Cross-validation vs bootstrapping for classification test (no clear conclusion, indifferent, but intersting papers? )

Answer 1

There are different ways to use the bootstrap for model validation. That might be leading to some confusion in both this and other pages on this site.

The method proposed by the OP and used in most of the links in the question and another answer here is to take a bootstrap sample, build the model on it, and evaluate its performance on the cases in the full data set that were omitted from the bootstrap sample. Repeat many times. That's often called the "out of bag" method.

That's not the method most consistent with the fundamental bootstrap principle, however: that taking bootstrap samples from the original full data set mimics the process of taking the original full data set from the underlying population.

Under that bootstrap principle, what you want to do is to evaluate a model from a bootstrap sample on the original full data set. That's most similar to how the model from the original full data might perform on application to the underlying population. Furthermore, comparing the performance of that model between its own bootstrap sample and on the original full data set provides an estimate of the optimism of the modeling process with respect to its performance in the underlying population. (If there has been some set of predictor-selection or similar steps involved in building the model, this method should involve all those steps for each bootstrap sample.)

Do the above on many bootstrap samples to get an averaged estimate of the optimism for whatever measure of model performance you want. You then might correct the original model developed from the full data set for its estimated optimism. That's called the "optimism bootstrap." Frank Harrell outlines and illustrate its use in this answer; it's implemented by tools in his rms package in R.

Harrell used those tools to compare several validation methods, and to evaluate how well they represent performance of logistic regression models on an underlying population, on this web page. In that simulation study, at least, "bootstrap is better than or equal to all cross-validation strategies tried." Also: "The relative comparison of validation methods depends in general on which index you are validating."

As you look through studies comparing "bootstrap" and other validation methods, make sure to know whether the authors used the "out of bag," the "optimism" bootstrap, or some other variant. If there was some form of model selection, ask whether all the modeling steps were repeated on each bootstrap sample. Pay close attention to the particular index of model performance evaluated.

Answer 2

TLDR/summary:

• Simulation studies have compared bootstrap and cross-validation methods for assessing generalization performance (i.e. estimated prediction accuracy for new data the model has not seen before).
• In general, there is no clear winner method that is best all of the time. However, repeated 5 or 10-fold CV and the bootstrap 632+ methods are often recommended, it seems.

Detailed answer:

Here are some notes from a few papers that compare resampling methods for assessing generalization performance:

Molinaro (2005). Prediction error estimation: a comparison of resampling methods. Bioinformatics, 21(15), 3301-3307.

• leave one out CV, 5- and 10-fold CV, and the .632+ bootstrap had the lowest mean square error.
• The .632+ bootstrap is quite biased in small sample sizes with strong signal-to-noise ratios
• Sort of seems to recommend 10-fold CV among the compared methods. Shows situation where .632+ boot is biased.

Iba, K. (2021). Re-evaluation of the comparative effectiveness of bootstrap-based optimism correction methods in the development of multivariable clinical prediction models. BMC Medical Research Methodology, 21, 1-14.

• Although this paper did not assess cross-validation, they compared 3 bootstrap methods, 2 of which are mentioned by the comments to your post.
• Compared Harrell's bias correction (i.e. the Efron and Gong optimism bootstrap method) and the .632 and .632+ estimators
• Under relatively large sample settings (roughly, events per variable ≥ 10), the three bootstrap-based methods were comparable and performed well. However, all three methods had biases under small sample settings, and the directions and sizes of biases were inconsistent.
• In general, Harrell’s and .632 methods had overestimation biases when event fraction become lager. .632+ method had a slight underestimation bias when event fraction was very small. Although the bias of the .632+ estimator was relatively small, its root mean squared error (RMSE) was comparable or sometimes larger than those of the other two methods, especially for the regularized estimation methods.
• In general, the three bootstrap estimators were comparable, but the .632+ estimator performed relatively well under small sample settings, except when the regularized estimation methods are adopted.
• Overall, IMO, there is not a clear winner from the 3 bootstrap methods in this paper. I guess the .632+ seems slightly better based on their results, but its hard to say.

Ch 4 of Kuhn, M. (2013). Applied predictive modeling.

K-fold CV:

• For k-fold CV, the choice of k is usually 5 or 10
• As k gets larger the bias of the technique becomes smaller (i.e., the bias is smaller for k = 10 than k = 5). In this context, the bias is the difference between the estimated and true values of performance.
• As k gets larger, the bias gets smaller but the variance gets greater
• From a practical viewpoint, larger values of k are more computationally burdensome. In the extreme, LOOCV is most computationally taxing because it requires as many model fits as data points and each model fit uses a subset that is nearly the same size of the training set.
• Molinaro (2005) found that leave-one-out and k =10-fold cross-validation yielded similar results, indicating that k = 10 is more attractive from the perspective of computational efficiency.
• Research indicates that** repeating k-fold crossvalidation** can be used to effectively increase the precision of the estimates while still maintaining a small bias.

Bootstrap:

• Discusses the "out of bag bootstrap method", i.e. fit a model to a bootstrap sample and predict on the samples that were omitted from that bootstrap sample (~ 1/3 of samples will be held out of each bootstrap sample when sampling with replacement)
• In general, out-of-bag bootstrap error rates tend to have less uncertainty/variance than k-fold cross-validation (Efron 1983). However, on average, 63.2 % of the data points the bootstrap sample are represented at least once, so out-of-bag bootstrap has bias similar to k-fold cross-validation when k ≈ 2. If the training set size is small, this bias may be problematic, but will decrease as the training set sample size becomes larger
• A few modifications of the simple bootstrap procedure have been devised to eliminate this bias: the .632 and .632+ methods, the latter performs better for smaller sample sizes