2
$\begingroup$

On this forum, I've read quite a few posts on the use of bootstrapping a statistic known as optimism when evaluating various models for their out of sample, predictive performance. I personally have not used it much, but I have found it quite interesting from the fact that there is no data splitting involved and that we use all available data when evaluating some statistic. I know Frank Harrell wrote about this method in his book, as well as Hastie et. al in the Elements of Statistical Learning.

For those who may not know, the optimism adjusted bootstrap technique is as follows:

  1. Fit a model on all of the data.
  2. Calculate the apparent error of this model (that is, the error the model demonstrates on the original dataset it has already been trained on)
  3. Create a large amount of bootstrap resamples.
  4. For every bootstrap resample, fit the model on that resample, calculate the apparent error for this model on the bootstrap resample it was trained on, and find the apparent error on the original dataset used in step 2).
  5. Calculate the optimism; using the model fit in step 4), find: apparent error using the bootstrap resample - apparent error using the original dataset.
  6. Take the average optimism from all of the bootstrap samples.
  7. Use step 2) observed error - step 5) average to get final estimate corrected for overfitting.

However, recently I've come across this very recent (as in December 2018) series of blog posts on r-bloggers that have seemingly found an apparent bias that regular cross validation does not.

Part 1 of this blog post
Part 2
Part 3
Part 4
Part 5

In these posts, the author disputes the usage of the technique and (seemingly) demonstrates that as the dimension of the dataset becomes large when compared to the number of observations, there is a systematic optimistic bias in the algorithm described above. Now clearly, the algorithm above leaks data (which is what the author of these blog posts claims is the reason for this behavior) but I am not too convinced by this logic as I would think that this behavior would then be present in situations where the dimension is much lower.

Harrell responded to these blog posts and his results actually seem to verify that indeed, the above algorithm seems to be vastly outperformed by cross validation for large dimensional data. In my opinion, it may even appear that the small performance benefits the bootstrapping algorithm may give on low dimensional data vs. cross validation may be severely outweighed by the large discrepancies in higher dimensions. Personally, I find myself working more and more with higher dimensional data where the number of candidate variables could be in the thousands.

Does anyone have any good explanation as to why this behavior exists (or is the blog poster's argument legit)? Has there been any explanation given since?

$\endgroup$
6
  • $\begingroup$ Two thoughts (before having read the blog posts in detail): 1. Data leaking present in low dimension/low complexity high n situations: sure it's there. But the prediction error is composed of bias (systematic error, e.g. due to underfitting) plus variance (e.g. due to model instability ≈ overfitting). So while it may be there all right already in low p high n situations, we won't be able to detect it between larger systematic error and possibly large random uncertainty of the estimate due to small absolute n_test. $\endgroup$ – cbeleites unhappy with SX Apr 7 '19 at 14:03
  • $\begingroup$ 2. As for the technique not splitting. While it doesn't distinguish training and test cases, the distinction of what is a case would still be crucial (altough the blog posts don't touch that issue): one important source of optimistic bias due to data leakage I meet in practice is that rows of the data set are not independent cases but related (think along the lines of repeated measurements). Such a structure in the data that necessitates careful thought about how to split for cross validation or out-of-bootstrap would mean for the procedure described here that the bootstrap estimate is approx. $\endgroup$ – cbeleites unhappy with SX Apr 7 '19 at 14:07
  • 1
    $\begingroup$ Coming from a field where high p low n is the standard situation, I think that there's a limit to what can be done in correcting overoptimism. In particular I think that it is very difficult to estimate overfitting in a situation where training error is zero, because we cannot detect any difference between a model with that just sufficient overfitting to make training error 0 from a model that does overfit even more wildy - they all have 0 training error. The optimism due to mixing (≈ 63.2 %) training with (≈ 37.8 %) test error in the optimism corrected bootstrap may still increase. $\endgroup$ – cbeleites unhappy with SX Apr 7 '19 at 14:28
  • 2
    $\begingroup$ I'd guess that establishing boundaries for the situations in which to use these methods would be valuable. E.g. both train and optimism-corrected estimates should be > 0 and not too different. (Too different: if there's, say, an order of magnitue in between them, I'd think one can reliably conclude there's a problem with overfitting, even though the optimism corrected bootstrap estimate may not be reliable). $\endgroup$ – cbeleites unhappy with SX Apr 7 '19 at 14:33
  • 1
    $\begingroup$ All that being said, I'll have to read the original paper - so far the advantage of the method (over out-of-bootstrap, or, if you like .632+-bootstrap or cross validation) isn't at all clear to me. $\endgroup$ – cbeleites unhappy with SX Apr 7 '19 at 14:34
1
$\begingroup$

Results from newer simulations with the code used can be found here: https://www.r-bloggers.com/part-6-how-not-to-validate-your-model-with-optimism-corrected-bootstrapping/. Zipped code and results can be found on the source website. Note that, the same bias can be seen using; 1) my implementation, 2) caret implementation.

It is interesting the random forest model gives much more bias even with more 'usual' dimensions, this confirms that the optimism corrected bootstrapping procedure has a fundamental problem with it which methods such as repeated cross validation do not have.

The bias depends on the dimensionality of the data and the type of model used, however, under a range of conditions it will always give a very positive result, even if the data is completely random.

Be warned of this method, if in doubt, run the code on this page yourself, it is a pretty straightforward experiment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.