This bootstrap-based internal validation procedure is implemented in the
validate function in Harrell's
rms package. It allows estimation of the 'optimism' inherent in a predictive accuracy measure derived from model training and testing on the same sample (i.e. the apparent accuracy). This optimism is estimated by taking bootstrap samples from the full data, and for each sample, one carries out the same model development procedure applied to the full data and then evaluates performance of the resulting model on the bootstrap sample it was developed on, and also the full sample. The difference between these performance estimates is then averaged over bootstrap samples, to obtain an estimate of the optimism, which is then subtracted from the apparent accuracy.
Now, I understand the point of the bootstrapping, to simulate the process of carrying out your model fitting/development procedure on a training sample, then testing it on an independent external validation sample. The difference in predictive accuracy gives an estimate of the performance drop one would expect moving from internal to external validation. The problem here is that each bootstrap sample overlaps quite substantially (on average, 63.2%) with the full sample, so optimism is underestimated, compared to what you'd get if you trained and tested your model on 2 genuinely independent samples (as you would in practice). Why not train the model on the bootstrap sample and test on the out-of-bag fraction (the data points that didn't make it into the bootstrap sample), rather than the full data? This way, you would be getting more honest estimates of optimism. Is there any inherent problem with this approach?