20
$\begingroup$

The bootstrap method has seen a great diffusion in the last years, I also use it a lot, especially because the reasoning behind is quite intuitive.

But that's one thing I don't understand. Why Efron chose to perform resample with replace instead of simply subsampling by randomly including or excluding single observations?

I think that random subsampling has one very good quality, that is represent ideally the real life situation in which the observations we have in our study are a subset of an hypothetical population. I don't see the advantage of having multiplied observations during resampling. In a real context no observation is similar to another, especially for complex multivariate situations.

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ resampling with resampling is done because that is the right thing to do, given the model. The model behind the bootstrap is to use nonparametric maximum likelihood to estimate the cumulative distribution function, then sampling independent observations from the estimated cumulative distribution function. Think about it---algoritmically, that is obtained by sampling by replacement from the original sample. $\endgroup$
    – kjetil b halvorsen ♦
    Sep 7, 2015 at 14:10

1 Answer 1

Reset to default
19
$\begingroup$

One way to understand this choice is to think of the sample at hand as being the best representation you have of the underlying population. You may not have the whole population to sample from any more, but you do have this particular representation of the population. A truly random re-sample from this representation of the population means that you must sample with replacement, otherwise your later sampling would depend on the results of your initial sampling. The presence of a repeated case in a particular bootstrap sample represents members of the underlying population that have characteristics close to those of that particular repeated case. Leave-one-out or leave-several-out approaches, as you suggest, can also be used but that's cross validation rather than bootstrapping.

I think this pretty much just puts into other words the comment from @kjetil_b_halvorsen

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ I understand the point. Making individual observations in a bootstrap sample independent from each other. In literature do exist methods based on subsampling, see Politis, Romano, Wolf. The use a fixed subset m of n, chosen without replacement. How do they avoid the pitfall you said before? In their case again I don't understand why they use a fixed size subsample instead of random subsample. $\endgroup$
    – Bakaburg
    Sep 7, 2015 at 15:08
  • 5
    $\begingroup$ Subsampling methods are trying to accomplish something different from the bootstrap. Those methods are seeking to select random subsets from the data sample rather than trying to emulate a new random sample from the underlying population. It's not that one or the other is wrong; they are different approaches that have particular strengths and weaknesses. $\endgroup$
    – EdM
    Sep 7, 2015 at 15:14
  • $\begingroup$ So maybe I should ask a new question regarding the difference between the two method in inference statistics. thanks! $\endgroup$
    – Bakaburg
    Sep 7, 2015 at 15:44
  • $\begingroup$ @Bakaburg see this question for a superb introduction into the literature on bootstrapping versus cross-validation (which is a particular type of subsampling). $\endgroup$
    – EdM
    Sep 7, 2015 at 16:24
  • $\begingroup$ @Bakaburg The bootstrap method is simulating the repeated independent drawing of random samples of size n (not a subset smaller than n) from a larger population. This means it's conceivable a random sample would contain a large number of extreme small or large values from the parent population which are often under-represented in our original sample. As EdM pointed out, resampling w/ replacement allows a single sample observation to "represent" multiple observations in the population which have similar values - it's a way to obtain a smooth approximation of the population distribution. $\endgroup$
    – RobertF
    Sep 4, 2018 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.