12
$\begingroup$

I think I understand how the fundamentals of bootstrapping work, but I'm not sure I understand how I can use bootstrapping for model selection or to avoid overfitting.

For model selection, for example, would you just choose the model that yields the lowest error (maybe variance?) across its bootstrap samples?

Are there any texts that discuss how to use bootstrapping for model selection or validation?

EDIT: See this thread, and the answer by @mark999 for more context behind this question.

|cite|improve this question
$\endgroup$
  • $\begingroup$ @suncoolsu If I have model A,B and C to choose from I would typically use either cross validation or bootstrapping to choose a model when 1) I am interested in prediction accuracy / ranking and 2) I dont have enough data for a hold out validation set. Why would this not be a good idea (and I know that nested validation is important for feature selection etc). $\endgroup$ – B_Miner Aug 19 '11 at 18:06
  • $\begingroup$ The answer by @mark999 in this thread suggests bootstrap validation as a solution to learning a model on the full dataset while still coping with overfitting. That answer is what inspired this question to a great extent, and the original question in that thread should also add context to this question. $\endgroup$ – Amelio Vazquez-Reina Aug 19 '11 at 18:29
  • 1
    $\begingroup$ I am sorry -- probably this is me being a statistician -- but I think cross-validation and bootstrap as two different things. Cross-validation is great and must be done (and bootstrap as well). But if you are in scenarios like choosing between A, B, C (only three models), BIC may be a better choice. As I said, the solution depends on the problem at hand and multiple approaches may be appropriate. $\endgroup$ – suncoolsu Aug 19 '11 at 18:48
  • 2
    $\begingroup$ AIC generally yields less underfitting than BIC. $\endgroup$ – Frank Harrell Aug 20 '11 at 15:09
13
$\begingroup$

First you have to decide if you really need model selection, or you just need to model. In the majority of situations, depending on dimensionality, fitting a flexible comprehensive model is preferred.

The bootstrap is a great way to estimate the performance of a model. The simplest thing to estimate is variance. More to your original point, the bootstrap can estimate the likely future performance of a given modeling procedure, on new data not yet realized.

If using resampling (bootstrap or cross-validation) to both choose model tuning parameters and to estimate the model, you will need a double bootstrap or nested cross-validation.

In general the bootstrap requires fewer model fits (often around 300) than cross-validation (10-fold cross-validation should be repeated 50-100 times for stability).

Some simulation studies may be found at http://biostat.mc.vanderbilt.edu/rms

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Wow, I didn't know that 10-fold CV should be repeated 50-100 times! I'll have to revisit my last project and try bootstrap testing instead. Love this website: I learn something every day! $\endgroup$ – Wayne Aug 20 '11 at 19:46
  • $\begingroup$ Thanks @Frank! Say I have a set of candidate models with the same # of parameters, are those with lower variance across the bootstrap estimations better candidates (assuming that the total loss or risk was the same for all of them) for fighting overfitting? $\endgroup$ – Amelio Vazquez-Reina Aug 20 '11 at 20:40
  • $\begingroup$ I wouldn't assume that but it's possible. $\endgroup$ – Frank Harrell Aug 20 '11 at 21:42
  • $\begingroup$ Great answer, thank you! I didn't know that bootstrap can also be used for model validation AND cross validation need to be repeated many times. I see another advantage of this method: cross validation requires the number of folds to be determined (subjectively) beforehand, typically 10, which is more or less heuristic rather than optimal. But while this is a great method, why it seems not as popular as cross validation? $\endgroup$ – SiXUlm Feb 21 at 0:17
  • $\begingroup$ Bootstrap model validation is fairly popular, but cross-validation has been around longer. But as you said there is some arbitrariness in the choice of # folds in c-v. $\endgroup$ – Frank Harrell Feb 21 at 17:22
3
$\begingroup$

Consider using the bootstrap for model averaging.

The paper below could help, as it compares a bootstrap model averaging approach to (the more commonly used?) Bayesian modeling averaging, and lays out a recipe for performing the model averaging.

Bootstrap model averaging in time series studies of particulate matter air pollution and mortality

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I would not recommend the bootstrap for model averaging in most cases. The bootstrap is best at telling you how one modeling procedure performs, rather than telling you how to create a new procedure. There are exceptions to this, though. $\endgroup$ – Frank Harrell Aug 20 '11 at 15:08
  • $\begingroup$ @Frank Harrell - Agreed. The paper I referenced applies to the area I work in sometimes and I have used the bootstrap for the scenario you stated: assessing a particular model's variability due to sampling error. But the uncertainty due to model selection itself is even harder to assess and the bootstrap model averaging approach could be useful as an aid, especially for practitioners like myself who lack the experience/background to reformulate problems for Bayesian model averaging. $\endgroup$ – Josh Hemann Aug 20 '11 at 16:08
  • 1
    $\begingroup$ No, I would say that the bootstrap is excellent for giving an assessment of the damage caused by not knowing the model in advance. That doesn't imply you should necessarily use the bootstrap to improve things, such as averaging over a set of uncertain models. If you should use the bootstrap in this way you would need a double bootstrap to get an honest assessment of performance of the averaged model. I should note that random forests are a form of model averaging using the bootstrap. $\endgroup$ – Frank Harrell Aug 20 '11 at 16:33
  • $\begingroup$ Good point about the double bootstrap. The authors of the paper I referenced have a follow up paper about this: Bootstrap-after-Bootstrap Model Averaging for Reducing Model Uncertainty in Model Selection for Air Pollution Mortality Studies $\endgroup$ – Josh Hemann Aug 20 '11 at 18:19
  • $\begingroup$ Good. Just remember that's often an overkill. It is often best to pose a subject-matter-driven full model, and to use shrinkage (penalization) if it overfits; but it's still one model. $\endgroup$ – Frank Harrell Aug 20 '11 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.