3
$\begingroup$

I understand that k-fold cross validation is a pessimistic model validator because it overestimates generalization error as less data is involved in training sets. Is bootstrapping called "optimistic" when it considers duplicates (replacement)? How this affects the generalization error?

And also, I understand both methods are useful to validate a model, so I do not know when I should use one over the other (pessimistic vs optimistic). Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Plain vanilla out-of-bootstrap validation is pessimistic just like cross validation (usually a bit more, actually): the bootstrapped training set is nominally the same size as the original - but contains copies. This doesn't amount to more training cases compared to the original data, and often/on average to somewhat less (so a copy doesn't seem to be as good as a really new case, which is fine).

There are varieties of the bootstrap that can be optimistically biased, depending on the data. .632-bootstrap tries to correct for the pessimistic bias by mixing 63.2 % of the pessimistic out-of-bootstrap with 16.8 % of the optimistic resampling error. If the models badly overfit, so have extremely low resampling error compared to out-of-bootstrap and true generalization error, this mix can end up being optimistic.

We did observe this for some extreme high-p-low-n situations: Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. & Sowa, M. G. Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).

From what we found then and what I've seen later on with other data (still spectroscopic, so pretty much always highly-correlated high p situations) I'd say that it is pretty much a matter of personal choice whether you do out-of-bootstrap or iterated/repeated cross validation. For my data, I just

  • avoid the .632-bootstrap because of the optimistic bias for our type of data, and .632+ isn't much different from vanilla oob.
  • make sure I have a reasonable number of iterations/repetitions for the cross validation.
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I'm confused. You are saying that out-of-bootstrap validation is pessimistic like cross-validation. What exactly do you mean by out-of-bootstrap validation (what type of validation method?). This Q was asking if vanilla bootstrap was optimistic which I think is correct, so one often corrects for that optism (0.632 is one way of doing so). Do you agree? $\endgroup$
    – Josh
    May 26, 2020 at 22:29
  • 1
    $\begingroup$ @Josh: with out-of-bootstrap I mean: bootstrap resample surrogate training sets and test each surrogate model with the cases left out. This estimate has a pessimistic bias (too hig error, too low accuracy, ...): on average the approximation "thinks" the model is worse than it actually is if the splitting procedure is correct. The model here is the model trained on the full data set, i.e. the model whose performance is approximated by the performance of the surrogate models. If you find an optimistic bias, in my experience the culprit is usually that the splitting was not done correctly... $\endgroup$
    – cbeleites unhappy with SX
    May 27, 2020 at 3:55
  • 1
    $\begingroup$ ... and then the splitting needs to be corrected before we can talk about further corrections like .632 (which corrects for the pessimism) or .632+ (which tries to detect whether the .632-corrected estimate became optimistic due to over-correcting the bootstrap/out-of-bootstrap pessimism, and corrects that optimistic bias). $\endgroup$
    – cbeleites unhappy with SX
    May 27, 2020 at 4:00
  • $\begingroup$ +1 Thanks - I think what confused me here is why you answered this question with a statement about the bias of the bootstrap surrogate in data that was not part of the bootstrap sample, since one often evaluates the performance of each bootstrap surrogate either in the bootstrap sample or in the full dataset. Furthermore, the question seems to be asking about optimistically biased estimates. But perhaps I'm missing the motivation of that comment in your answer? $\endgroup$
    – Josh
    May 27, 2020 at 4:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.