10
$\begingroup$

I'm learning about bootstrapping as a means of estimating the variance of a sample statistic. I have one basic doubt.

Quoting from http://web.stanford.edu/class/psych252/tutorials/doBootstrapPrimer.pdf:

• How many observations should we resample? A good suggestion is the original sample size.

How can we resample as many observations as in the original sample?
If I have a sample size of 100, and I'm trying to estimate the variance of the mean. How can I obtain multiple bootstrap samples of size 100 from a total sample size of 100? Only 1 bootstrap sample would be possible in this case which would be equivalent to the original sample right?

I'm obviously misunderstanding something very basic. I understand that the number of ideal bootstrap samples is always infinite, and to determine the number of bootstrap samples necessary for my data I'd have to test for convergence keeping my required precision in mind.
But I'm really confused about what should be the size of each individual bootstrap sample.

Improve this question
$\endgroup$
4
  • 8
    $\begingroup$ The top of p. 3, and the illustrations there, clearly and prominently stipulate that resampling is with replacement. $\endgroup$
    – whuber ♦
    Nov 18, 2016 at 15:04
  • $\begingroup$ But if my bootstrap sample size is the same as the total number of observations I have, what do I replace with? $\endgroup$
    – user1265125
    Nov 18, 2016 at 15:08
  • $\begingroup$ Simplified example - so if I have 4,1,3,7,5 as my sample set. How can I create multiple bootstrap samples of size 5? The only size 5 bootstrap sample will be 4,1,3,7,5 i.e. the original sample set. $\endgroup$
    – user1265125
    Nov 18, 2016 at 15:10
  • 1
    $\begingroup$ Oh wait I understood - "• To simulate a sampling distribution, we can just take repeated random samples from this “population” made up of many copies of the sample" $\endgroup$
    – user1265125
    Nov 18, 2016 at 15:11

2 Answers 2

Reset to default
24
$\begingroup$

Bootstrap is conducted by sampling with replacement. It seems that the term "with replacement" is unclear for you. As noted by whuber, illustration of sampling with replacement is given on p. 3 of the paper you refer to (reproduced below).

Illustration of sampling with replacement

(source: http://web.stanford.edu/class/psych252/tutorials/doBootstrapPrimer.pdf)

The general idea of sampling with replacement is that any case can be sampled multiple times (green marble on the first image above; blue and violet marbles on the last picture). If you want to imagine yourself this process, think of a bowl filled with colorful marbles. Say that you want to draw some number of marbles from this bowl. If you sampled without replacement, then you would be simply taking the marbles out of the bowl and putting the sampled ones aside. If you sampled with replacement, then you would be sampling the marbles one-by-one, by taking single marble out of the bowl, signing down it's color in your notebook and then returning it back to the bowl. So when sampling with replacement the same marble can be sampled multiple times.

So when sampling without replacement, you can sample only $n$ marbles out of the bowl containing $n$ marbles, while in case of sampling with replacement you can sample any number of marbles (even greater then $n$) from the finite population. If you sampled $n$ out of $n$ marbles without replacement you would end up with exactly the same sample but in shuffled order. If you sampled $n$ out of $n$ marbles with replacement, each time you can possibly sample a different combination of marbles.

There is $n \choose k$ ways of sampling without replacement $k$ cases out of population of size $ n$ and $n+k-1 \choose k$ ways of sampling with replacement. If you want to read more about the math behind it, you can check the 2.1. Combinatorics chapter of Introduction to Probability online handbook by Hossein Pishro-Nik. There is also a handy cheatsheet on WolframMathWorld page.

Improve this answer
$\endgroup$
2
$\begingroup$

How many observations should we re-sample? A good suggestion is the original sample size.

When the original sample size is too large and you don't want/cannot train a model on the full dataset, the "good suggestion" is not so good.

PS: I wanted to add this as a comment to the question but I am not allowed to add any comment...

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Why do you want to add this suggestion? If this is due to datasets too large for regular computational efforts, that is a relevant practical issue, but it does not really apply to the theory of bootstrapping that was questioned here. Moreover, this was about 'estimating the variance of a sample statistic'. Is that really related to generally training a model? (NB. not to be rude, I understand you cannot yet post comments, but that does not discharge you from providing a relevant answer when posting as such. You have to be a lot more clear, OR post your own question) $\endgroup$
    – IWS
    Dec 4, 2017 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.