0
$\begingroup$

On this forum, I've read quite a few posts on the use of bootstrapping a statistic known as optimism when evaluating various models for their out of sample, predictive performance. I personally have not used it much, but I have found it quite interesting from the fact that there is no data splitting involved and that we use all available data when evaluating some statistic. I know Frank Harrell wrote about this method in his book, as well as Hastie et. al in the Elements of Statistical Learning.

For those who may not know, the optimism adjusted bootstrap technique is as follows:

  1. Fit a model on all of the data.
  2. Calculate the apparent error of this model (that is, the error the model demonstrates on the original dataset it has already been trained on)
  3. Create a large amount of bootstrap resamples.
  4. For every bootstrap resample, fit the model on that resample, calculate the apparent error for this model on the bootstrap resample it was trained on, and find the apparent error on the original dataset used in step 2).
  5. Calculate the optimism; using the model fit in step 4), find: apparent error using the bootstrap resample - apparent error using the original dataset.
  6. Take the average optimism from all of the bootstrap samples.
  7. Use step 2) observed error - step 5) average to get final estimate corrected for overfitting.

However, recently I've come across this very recent (as in December 2018) series of blog posts on r-bloggers that have seemingly found an apparent bias that regular cross validation does not.

Part 1 of this blog post
Part 2
Part 3
Part 4
Part 5

In these posts, the author disputes the usage of the technique and (seemingly) demonstrates that as the dimension of the dataset becomes large when compared to the number of observations, there is a systematic optimistic bias in the algorithm described above. Now clearly, the algorithm above leaks data (which is what the author of these blog posts claims is the reason for this behavior) but I am not too convinced by this logic as I would think that this behavior would then be present in situations where the dimension is much lower.

Harrell responded to these blog posts and his results actually seem to verify that indeed, the above algorithm seems to be vastly outperformed by cross validation for large dimensional data. In my opinion, it may even appear that the small performance benefits the bootstrapping algorithm may give on low dimensional data vs. cross validation may be severely outweighed by the large discrepancies in higher dimensions. Personally, I find myself working more and more with higher dimensional data where the number of candidate variables could be in the thousands.

Does anyone have any good explanation as to why this behavior exists (or is the blog poster's argument legit)? Has there been any explanation given since?

$\endgroup$
  • $\begingroup$ Two thoughts (before having read the blog posts in detail): 1. Data leaking present in low dimension/low complexity high n situations: sure it's there. But the prediction error is composed of bias (systematic error, e.g. due to underfitting) plus variance (e.g. due to model instability ≈ overfitting). So while it may be there all right already in low p high n situations, we won't be able to detect it between larger systematic error and possibly large random uncertainty of the estimate due to small absolute n_test. $\endgroup$ – cbeleites Apr 7 at 14:03
  • $\begingroup$ 2. As for the technique not splitting. While it doesn't distinguish training and test cases, the distinction of what is a case would still be crucial (altough the blog posts don't touch that issue): one important source of optimistic bias due to data leakage I meet in practice is that rows of the data set are not independent cases but related (think along the lines of repeated measurements). Such a structure in the data that necessitates careful thought about how to split for cross validation or out-of-bootstrap would mean for the procedure described here that the bootstrap estimate is approx. $\endgroup$ – cbeleites Apr 7 at 14:07
  • 1
    $\begingroup$ Coming from a field where high p low n is the standard situation, I think that there's a limit to what can be done in correcting overoptimism. In particular I think that it is very difficult to estimate overfitting in a situation where training error is zero, because we cannot detect any difference between a model with that just sufficient overfitting to make training error 0 from a model that does overfit even more wildy - they all have 0 training error. The optimism due to mixing (≈ 63.2 %) training with (≈ 37.8 %) test error in the optimism corrected bootstrap may still increase. $\endgroup$ – cbeleites Apr 7 at 14:28
  • 1
    $\begingroup$ I'd guess that establishing boundaries for the situations in which to use these methods would be valuable. E.g. both train and optimism-corrected estimates should be > 0 and not too different. (Too different: if there's, say, an order of magnitue in between them, I'd think one can reliably conclude there's a problem with overfitting, even though the optimism corrected bootstrap estimate may not be reliable). $\endgroup$ – cbeleites Apr 7 at 14:33
  • 1
    $\begingroup$ All that being said, I'll have to read the original paper - so far the advantage of the method (over out-of-bootstrap, or, if you like .632+-bootstrap or cross validation) isn't at all clear to me. $\endgroup$ – cbeleites Apr 7 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.