# Is bootstrap variance (based on mean) same as standard error?

If one uses the data mean $$\bar{X}$$ to estimate the population mean of a Normal($$\mu$$, $$\sigma^2$$) distribution of $$X_i$$'s, would the bootstrap variance be the same as the estimate's variance? aka $$\sigma^2/n$$?

I know that the mean of the boostrap would be $$\bar{X}$$, but does the variance calculation change or am I here on a loop?

No, it wouldn't. They are two different calculations of the standard error of the sample mean, and as such will result in different estimates of the standard deviation of the sample mean.

Here is an example:

# Observations
x  <- rnorm(1000)

# True value
sd_true <- 1/sqrt(1000)

# Estimate based on sample std. deviation
se_xbar <- sqrt(var(x)/1000)

# Estimate based on bootstrap replications
xbar_boot <- rep(0,10000)
for (j in 1:length(xbar_boot)) {
z <- sample(x, 1000, replace=TRUE)
xbar_boot[j] <- mean(z)
}

se_xbar_boot <- sd(xbar_boot)


... and the results ...

> sd_true
 0.03162278
> se_xbar
 0.03122161
> se_xbar_boot
 0.03111651


As you can see, the three calculations give similar, but nonetheless different, values. As the number of observations in your original sample $$\to \infty$$, the estimate based on the sample standard deviation will converge to the true value, but in order for the bootstrap to converge to the true value, the number of bootstrap samples must also $$\to \infty$$.

The bootstrap standard error is close to the sample standard error. There are a few caveats.

If the number of bootstrap replication $$B$$ is very large they will be arbitrarily close but never identical. There is always noise that comes from the resampling process. If $$B$$ is too small the standard errors might differ substantially.

• This result holds generally, and does not depend on $$X$$ being normally distributed.
• I will show why by breaking down what the bootstrap is, and why $$B$$ needs to be large.
• It is actually reassuring that the bootstrap standard error is similar to the simple case. The true usefulness of the bootstrap is for more complex models where an explicit formula for the standard errors is difficult to obtain. For example, in two-step estimators or nonlinear models.
• If $$B,n \to \infty$$ both converge to the population standard error.

Definitions: What is the bootstrap?

Let $$X_{bi}$$ be a bootstrap draw from the sample $$\mathcal{X}_n = \{X_1,\ldots,X_n\}$$. Formally, the bootstrap draw is defined as $$X_{bi} = \sum_{j=1}^n X_j S_{bij}$$ where the $$S_{bi} = \{S_{bi1},\ldots,S_{bin}\}$$ is a vector of indicators that take values in $$\{0,1\}$$, follow a multinomial distribution and are independent across $$i$$. It can be shown that $$\mathbb{V}(S_{bij}) = \frac{n-1}{n^2}$$ and $$Cov(S_{bij},S_{bik}) = -\frac{1}{n^2}$$ for $$j \ne k$$. The indicators for a given $$i$$ are negatively correlated because sampling observation $$j$$ means that $$k$$ was not chosen. The fact that $$S_{bi}$$ are independent allows for sampling with replacement, and means that $$X_j$$ can be drawn more than once. We also assume that the researcher generates $$S_{bi}$$ to be independent of $$\mathcal{X}_n$$.

We say that a bootstrap sample is $$\mathcal{X}_{bn} = \{X_{b1},\ldots,X_{bn}\}$$. The bootstrap mean for replication $$b$$ is $$\bar{X}_b = \frac{1}{n}\sum_{i=1}^n X_{bi}$$ We can define the bootstrap standard error as $$\hat{\sigma}^2_{B} = \frac{1}{B}\sum_{b=1}^B \bar{X}_b^2 - \left( \frac{1}{B}\sum_{b=1}^B \bar{X}_b \right)^2$$

Why should we have many bootstrap replications?

Suppose that we hold $$\mathcal{X}_n$$ fixed, then all the randomness in $$\bar{X}_b$$ comes from the indicators. Therefore as $$B \to \infty$$, we can apply the weak law of large numbers to show that $$\hat{\sigma}^2_{B} \mid \mathcal{X}_n \to^{p} \mathbb{V}(\bar{X}_b \mid \mathcal{X}_n) \qquad \text{ as }B \to \infty$$ Keeping $$\mathcal{X}_n$$ fixed and increasing $$B$$ allows us to understand what happens to our bootstrap estimates when we increase $$B$$. It suggests that bootstrap standard errors must stabilize as we increase the number of replications. I've heard this referred coloquially as convergence in the "bootstrap world". It is important to emphasize that where are considering randomness of $$S_{bij}$$ while completely ignoring any randomness in $$X_j$$.

This is merely a numeric result related to the process of simulating $$S_{bij}$$.

Why are the standard errors similar when $$B \to \infty$$?

It all boils down to calculating $$\mathbb{V}(\bar{X}_b \mid \mathcal{X}_n)$$. This turns out to be a much simpler problem than the initial one we started with. Since the bootstrap indicators are i.i.d. across $$b$$ \begin{align*} \mathbb{V}(\bar{X}_b \mid \mathcal{X}_n) &= \frac{1}{n^2}\sum_{i=1}^n \mathbb{V}(X_{bi} \mid \mathcal{X}_n) \\ &= \frac{1}{n}\mathbb{V}(X_{bi} \mid \mathcal{X}_n)\\ \end{align*} Furthermore, we can plug in the above expression for $$X_{bi}$$ to show that \begin{align*} \mathbb{V}(\bar{X}_b \mid \mathcal{X}_n) &= \frac{1}{n}\mathbb{V}(X_{bi} \mid \mathcal{X}_n)= \frac{1}{n} \mathbb{V}\left( \sum_{j=1}^n X_j S_{bij} \mid \mathcal{X}_n \right) \\ &= \frac{1}{n}\sum_{j=1}^n X_j^2 \mathbb{V}(S_{bij}) + \frac{1}{n}\sum_{j=1}^n\sum_{k \ne j} X_jX_k Cov(S_{bij},S_{bil}) \\ &= \frac{1}{n}\left(\frac{n-1}{n^2} \right)\sum_{i=1}^n X_i^2 - \frac{1}{n}\left(\frac{1}{n^2}\right) \sum_{j=1}^n\sum_{j \ne k}X_jX_k \\ \end{align*} By rearranging the equation it is possible to show that $$\frac{1}{n^2}\sum_{j=1}^n\sum_{j \ne k}X_jX_k = \bar{X}^2 - \frac{1}{n^2} \sum_{j=1}^n X_j^2$$, where $$\bar{X}$$ is the sample mean. This implies that some terms cancel out after some tedious but straightforward algebra. The final result is \begin{align*}\mathbb{V}(\bar{X}_b \mid \mathcal{X}_n) &=\frac{1}{n}\left(\frac{1}{n} \sum_{i=1}^n X_i^2 - \bar{X}^2 \right) \\ &= \frac{1}{n}\hat{\sigma}^2 \end{align*} This shows that the bootstrap standard error indeed converges to the sample standard error as $$B \to \infty$$, keeping $$n$$ fixed.

• It is useful to distinguish the bootstrap, which is defined in terms of the entire distribution of all possible resamples, from its practical implementation, which samples randomly from that distribution. The answer for the bootstrap is simple and clear, while the answer for the sampling (that is, "$B\to\infty$") is an immediate consequence of laws of large numbers.
– whuber
Oct 21, 2021 at 16:22