# Variance of a bootstrap sample mean

I need help with calculating the variance of a bootstrap sample mean. I want to answer the following question (question 15.4 in The Elements of Statistical Learning).

Suppose $$x_i$$, $$i=1, \ldots , N$$ are iid $$(\mu, \sigma^2)$$. Let $$\bar{x}^\star_1$$ and $$\bar{x}^\star_2$$ be two bootstrap realizations of the sample mean. Show that the sampling correlation $$corr(\bar{x}^\star_1,\bar{x}^\star_2)=\frac{n}{2n-1} \approx 50\%$$. Along the way, derive $$\bar{x}^\star_1$$ and the variance of the bagged mean $$\bar{x}_{bag}$$. Here, $$\bar{x}$$ is a linear statistic; bagging produces no reduction in variance for linear statistic.

This is the solution that I have so far:

The sample mean $$\bar{x}$$ is given by

$$\bar{x}=\frac{1}{N} \sum_{i=1}^N x_i,$$

and the bootstrap sample mean $$\bar{x}^\star_1$$ is computed from a resampled dataset, the same applies for $$\bar{x}^\star_2$$. The sampling correlation between $$\bar{x}^\star_1$$ and $$\bar{x}^\star_2$$ is given by

$$corr(\bar{x}^\star_1,\bar{x}^\star_2)= \frac{Cov(\bar{x}^\star_1,\bar{x}^\star_2)}{\sqrt{Var(\bar{x}^\star_1)Var(\bar{x}^\star_2)}},$$

so, we need to begin with computing the covariance term $$Cov(\bar{x}^\star_1,\bar{x}^\star_2)$$ and the respective variance terms in order to calculate their correlation. We start by denoting that

$$cov(\bar{x}_1^\star, \bar{x}_2^\star)=\frac{\sigma^2}{n}$$

since $$E[\bar{x}_1^\star]=E[\bar{x}_2^\star]$$ which is the population mean. Now we want to calculate the variances.

However, this is where I get stuck. How do one calculate the variance of a bootstrap realization sample mean?

My first thought was that since the bootstrap sample is obtained by resampling with replacement, the variance of the sample mean for a bootstrap sample is the same as the variance of the original sample mean, i.e. $$\sigma^2/n$$, but this cannot be correct since this would not obtain $$corr(\bar{x}^\star_1,\bar{x}^\star_2)=\frac{n}{2n-1} \approx 50\%$$.

Any help would be appreciated in order to help me see what it is that I am missing here!

Update:

I (hopefully) solved it using the following definition

$$Var(\bar{x}_i^\star)=\frac{1}{n^2}\left[ \sum_{i=1}^n Var(x_i) + 2\sum_{i=1}^{n-1}\sum_{j=i+1}^n Cov(x_i^\star,x_j^\star)\right] = \frac{1}{n^2}\left[n\sigma^2+n(n-1)\frac{\sigma^2}{n}\right]$$

and obtained the variance $$\frac{\sigma^2(2n-1)}{n^2}$$. Note that $$x_i^\star$$ and $$x_j^\star$$ are resamples of the original dataset.

From this, I could insert the answers into the formula for correlation and prove that $$corr(\bar{x}^\star_1,\bar{x}^\star_2)=\frac{n}{2n-1} \approx 50\%$$. What do you think about this solution? Do you think it is correct (even though I did not use the bagged mean to prove it)?

I would do this by conditioning on the sample.

Write $$s^2=\frac{1}{n}\sum_{i=1}^n (x_i-\bar x)^2$$, so $$s^2$$ is the variance of the empirical distribution of the sample. Also, write $$\mathrm{var}_P$$ and $$\mathrm{var}_B$$ for the variance under sampling of the original $$x$$ and the variance under bootstrap resampling. All the bootstrap resamples share the same initial sample but have independent resampling.

We have

• $$\mathrm{var}_P[\bar x]=\sigma^2/n$$
• $$\mathrm{var}_B[\bar x_1^*]=s^2/n$$ (because it's just iid sampling from the empirical distribution)
• $$\mathrm{var}_B[\bar x_2^*]=s^2/n$$ (ditto)

Now, the total variance of $$\bar x_1^*$$ is given by the conditional variance formula $$\mathrm{var}_{PB}[\bar x_1^*]=E_P[\mathrm{var}_B[\bar x_1^*]]+\mathrm{var}_P[E_B[\bar x_1^*]]=E_P[s^2/n]+\mathrm{var}_P[\bar x]=\frac{n-1}{n}\frac{s^2}{n}+\sigma^2/n$$

Two bootstrap means share the sampling components of the variance but have independent resampling components. $$\mathrm{cov}[\bar x_1^*,\bar x_2^*]= \sigma^2/n$$ so $$\mathrm{corr}[\bar x_1^*,\bar x_2^*]=\frac{\mathrm{cov}[\bar x_1^\star,\bar x_2^\star]}{\sqrt{\mathrm{var}[\bar x_1^\star]\mathrm{var}[\bar x_2^\star]}}=\frac{\sigma^2/n}{\frac{n-1}{n}\frac{s^2}{n}+\sigma^2/n}\approx 1/2$$

And if you bag $$m$$ times you reduce the resampling component by a factor of $$m$$, because they are all conditionally independent: $$\mathrm{var}_{PB}[\bar x_\textrm{bag}]=\sigma^2/n+\frac{1}{m}\frac{n-1}{n}\frac{s^2}{n}$$ which gets closer to $$\sigma^2/n$$ as $$m$$ increases.