1
$\begingroup$

I understand that k-fold cross validation is a pessimistic model validator because it overestimates generalization error as less data is involved in training sets. Is bootstrapping called "optimistic" when it considers duplicates (replacement)? How this affects the generalization error?

And also, I understand both methods are useful to validate a model, so I do not know when I should use one over the other (pessimistic vs optimistic). Thanks!

$\endgroup$
1
$\begingroup$

Plain vanilla out-of-bootstrap validation is pessimistic just like cross validation (usually a bit more, actually): the bootstrapped training set is nominally the same size as the original - but contains copies. This doesn't amount to more training cases compared to the original data, and often/on average to somewhat less (so a copy doesn't seem to be as good as a really new case, which is fine).

There are varieties of the bootstrap that can be optimistically biased, depending on the data. .632-bootstrap tries to correct for the pessimistic bias by mixing 63.2 % of the pessimistic out-of-bootstrap with 16.8 % of the optimistic resampling error. If the models badly overfit, so have extremely low resampling error compared to out-of-bootstrap and true generalization error, this mix can end up being optimistic.

We did observe this for some extreme high-p-low-n situations: Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. & Sowa, M. G. Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).

From what we found then and what I've seen later on with other data (still spectroscopic, so pretty much always highly-correlated high p situations) I'd say that it is pretty much a matter of personal choice whether you do out-of-bootstrap or iterated/repeated cross validation. For my data, I just

  • avoid the .632-bootstrap because of the optimistic bias for our type of data, and .632+ isn't much different from vanilla oob.
  • make sure I have a reasonable number of iterations/repetitions for the cross validation.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.