On this forum, I've read quite a few posts on the use of bootstrapping a statistic known as optimism when evaluating various models for their out of sample, predictive performance. I personally have not used it much, but I have found it quite interesting from the fact that there is no data splitting involved and that we use all available data when evaluating some statistic. I know Frank Harrell wrote about this method in his book, as well as Hastie et. al in the Elements of Statistical Learning.
For those who may not know, the optimism adjusted bootstrap technique is as follows:
- Fit a model on all of the data.
- Calculate the apparent error of this model (that is, the error the model demonstrates on the original dataset it has already been trained on)
- Create a large amount of bootstrap resamples.
- For every bootstrap resample, fit the model on that resample, calculate the apparent error for this model on the bootstrap resample it was trained on, and find the apparent error on the original dataset used in step 2).
- Calculate the optimism; using the model fit in step 4), find: apparent error using the bootstrap resample - apparent error using the original dataset.
- Take the average optimism from all of the bootstrap samples.
- Use step 2) observed error - step 5) average to get final estimate corrected for overfitting.
However, recently I've come across this very recent (as in December 2018) series of blog posts on r-bloggers that have seemingly found an apparent bias that regular cross validation does not.
In these posts, the author disputes the usage of the technique and (seemingly) demonstrates that as the dimension of the dataset becomes large when compared to the number of observations, there is a systematic optimistic bias in the algorithm described above. Now clearly, the algorithm above leaks data (which is what the author of these blog posts claims is the reason for this behavior) but I am not too convinced by this logic as I would think that this behavior would then be present in situations where the dimension is much lower.
Harrell responded to these blog posts and his results actually seem to verify that indeed, the above algorithm seems to be vastly outperformed by cross validation for large dimensional data. In my opinion, it may even appear that the small performance benefits the bootstrapping algorithm may give on low dimensional data vs. cross validation may be severely outweighed by the large discrepancies in higher dimensions. Personally, I find myself working more and more with higher dimensional data where the number of candidate variables could be in the thousands.
Does anyone have any good explanation as to why this behavior exists (or is the blog poster's argument legit)? Has there been any explanation given since?