Choosing the number of bootstrap resamples

Say we have the following sample and we are trying to estimate the variance of the sample mean of the population.

X = [0, -1, 2, 10, -3]


If I take an increasing number of bootstrap (e.g. 100, 1000, 10,000, 100,0000), the bootstrap distribution of the sample mean gets narrower and narrower. In other words, I can make the bootstrap distribution arbitrarily narrow.

Given the above, what's the point of using a different number of bootstrap samples here? And wouldn't this mean that I could make the standard error of the mean arbitrarily small by just taking more and more bootstrap samples? What am I missing?

Please note that I am doing a proper bootstrap sample in the sense that each bootstrap sample has the same exact size as the original sample (input array).

• What are you asking about? To what quantity does "the distribution" refer in this question? – whuber Mar 17 '17 at 13:12
• Thanks @whuber By "the distribution" above I am referring to the distribution we get from collecting the mean from each individual bootstrap sample. By "a bootstrap sample" here I am referring to a single sample with replacement from the original sample and of the same size. That is, the distribution of 10,000 means we get from the process. – Josh Mar 17 '17 at 13:15
• The point about the mean is crucial to understanding the question. Please see stats.stackexchange.com/search?q=sampling+distribution+mean for several thousand references to this phenomenon, then take your pick of suitable answers. Another way to address your situation is to note that the only bootstrap samples you can take must be of the same size as your dataset: none of the others are bootstrap samples. See stats.stackexchange.com/questions/tagged/bootstrap?sort=votes. – whuber Mar 17 '17 at 13:21
• @whuber each sample (with replacement) that I take has the exact same size as my original sample, but I can still take multiple such samples, right? e.g. if my input array is [0, -1, 2, 5], and I take three bootstrap samples, they may be: [0,2,2,-1], [-2,0,0,5] and [5,-1,0,2] (each of the same size as the input array). Now, rather than three, I could also take 100, or 10K bootstrap samples right? So a question here for me is, if I use this approach to say, estimate the variance of a mean estimator here, how can I decide how many bootstrap samples to take? – Josh Jul 31 '17 at 2:09

A bootstrap sample is usually taken to mean that the sample size of the resample is equal to the original sample size. What you are doing is to take resamples from the original sample with larger and larger (re)sample sizes. There is no reason to believe that this will represent the properties of the (original) sampling from the study population.

Say you are interested in the mean of some unknown distribution $$F$$ (on the real line, to make example specific). The mean (assuming it exists ) $$\mu$$ of the distribution $$F$$ is given by $$\mu(F) = \int_{-\infty}^\infty x \; dF(x)$$ where the integral is a Stieltjes integral. If $$F$$ is the distribution of some continuous random variable with density $$f(x) =F'(x)$$ this is the usual integral $$\int x f(x) \; dx$$ but it also includes the discrete case. The point of writing the expectation in this unusual way is that we can see that the expectation is a functional of the distribution $$F$$, and also that it unifies the treatment of continuous/discrete cases.

Now we get a sample $$x_1, x_2, \dotsc, x_N$$ from $$F$$, and the idea behind bootstrapping is that we represent the distribution $$F$$ with the sample, and investigates sampling properties of estimators of $$\mu$$ by resampling from the sample. This makes clear that we need to assume that the sample is reasonably representative of $$F$$!, so we cannot expect this to work well with too small samples.

Now, our sample size was $$N$$, so we want properties of estimators of $$\mu$$ based on a sample of size $$N$$. Suppose we take resamples of size $$n$$ (possibly with $$n \not = N$$). Our resamples is a stand-in for samples from $$F$$ (that is the whole point with bootstrapping!). Suppose $$F$$ also has existing variance $$\sigma^2$$, and we estimate $$\mu$$ by the empirical mean $$\bar{x}=\frac{1}{N}\sum_i x_i=\int_{-\infty}^\infty x \;d\hat{F}_N(x)$$ where $$\hat{F}_n(x)$$ is the empirical distribution function at $$x$$. Then the variance of this estimator will be $$\sigma^2/N$$. Lets say we do resampling but with resamples of size $$n$$. Then the empirical mean based on this resamples will have variance $$\sigma^2(\hat{F}_N)/n$$ where $$\sigma^2(\hat{F}_N)$$ is the variance based on the sample. If this empirical variance is a good estimator of $$\sigma^2$$, this will be approximately $$\sigma^2/n$$. If $$n$$ is different from $$N$$, this cannot be a good representation of the variance of $$\bar{x}$$, so will not tell you about the real uncertainty in $$\bar{x}$$ as an estimator of $$\mu$$.

EDIT


To clarify, the error in the results when using bootstrapping can be decomposed in the sampling error (due to only taking $$N$$ observations), and the bootstrap error (due to only taking $$n < \infty$$ resamples). By increasing $$n$$ we can reduce the later, but not the former.

Sometimes one is deliberately using a bootstrap sample size different from the original. See Can we use bootstrap samples that are smaller than original sample?, Subsample bootstrapping

• Thanks, in my post I am keeping the size of each resample constant and equal to the size of the original array. What I was noticing was that the more resamples I take, the narrower the distribution of means of each resample gets as well. I am assuming this is expected but confused about (1) why, and (2) how the number of resamples should inform inference. – Josh Apr 19 '17 at 12:10
• Hi @kjetil this is extremely helpful. Thanks again. I updated the OP to clarify my question as per my comment above. I would be curious to know your thoughts on it. – Josh Aug 1 '17 at 14:57
• Why the downvote? An explication could be helpful, it could lead to a better answer ... – kjetil b halvorsen Dec 15 '20 at 13:42