This bootstrap-based internal validation procedure is implemented in the validate function in Harrell's rms package. It allows estimation of the 'optimism' inherent in a predictive accuracy measure derived from model training and testing on the same sample (i.e. the apparent accuracy). This optimism is estimated by taking bootstrap samples from the full data, and for each sample, one carries out the same model development procedure applied to the full data and then evaluates performance of the resulting model on the bootstrap sample it was developed on, and also the full sample. The difference between these performance estimates is then averaged over bootstrap samples, to obtain an estimate of the optimism, which is then subtracted from the apparent accuracy.

Now, I understand the point of the bootstrapping, to simulate the process of carrying out your model fitting/development procedure on a training sample, then testing it on an independent external validation sample. The difference in predictive accuracy gives an estimate of the performance drop one would expect moving from internal to external validation. The problem here is that each bootstrap sample overlaps quite substantially (on average, 63.2%) with the full sample, so optimism is underestimated, compared to what you'd get if you trained and tested your model on 2 genuinely independent samples (as you would in practice). Why not train the model on the bootstrap sample and test on the out-of-bag fraction (the data points that didn't make it into the bootstrap sample), rather than the full data? This way, you would be getting more honest estimates of optimism. Is there any inherent problem with this approach?

  • 1
    $\begingroup$ The out of bag procedure is not as efficient. The reason the bootstrap works as a bias estimator is that the difference between a fit from a sample with replacement and a fit on the whole sample estimates the difference between a fit on the whole sample and a fit on a new sample. Super-overfitting - regular overfitting = regular overfitting - no overfitting. $\endgroup$ – Frank Harrell May 15 '18 at 13:40
  • $\begingroup$ @FrankHarrell Thank you for the response, that really clears things up for me. $\endgroup$ – SethCamd May 21 '18 at 8:06
  • $\begingroup$ @FrankHarrell is there a simple answer why cross-validation on the out-of-bag sample is wrong (at least, it's not common practice)? $\endgroup$ – Agile Bean Feb 10 at 8:45
  • $\begingroup$ What is the principle behind it? More importantly, it is not how the bootstrap was intended to work and is not efficient. The simulations presented here are the type of simulations you would need to do to show that the out of bag method does not lose efficiency in terms of mean squared error. In the above report you'll see some results on modifications of the boostrap (.632 methods) that are related to this discussion. $\endgroup$ – Frank Harrell Feb 10 at 12:44
  • 1
    $\begingroup$ Use the term "cross-validation" exactly as it was intended. CV does not involve the bootstrap, uses a random, say, 9/10 of the data to develop a model, evaluates its performance on the remaining 1/10, averages that over the 10 splits, then averages all of that over as many replications as you have time for (up to 100 repeats of 10-fold CV). Repeated 10-fold CV will work over a wide variety of situations, handling p > n where the bootstrap is too biased. $\endgroup$ – Frank Harrell Feb 12 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.