1
$\begingroup$

I need help with calculating the variance of a bootstrap sample mean. I want to answer the following question (question 15.4 in The Elements of Statistical Learning).

Suppose $x_i$, $i=1, \ldots , N$ are iid $(\mu, \sigma^2)$. Let $\bar{x}^\star_1$ and $\bar{x}^\star_2$ be two bootstrap realizations of the sample mean. Show that the sampling correlation $corr(\bar{x}^\star_1,\bar{x}^\star_2)=\frac{n}{2n-1} \approx 50\%$. Along the way, derive $\bar{x}^\star_1$ and the variance of the bagged mean $\bar{x}_{bag}$. Here, $\bar{x}$ is a linear statistic; bagging produces no reduction in variance for linear statistic.

This is the solution that I have so far:

The sample mean $\bar{x}$ is given by

$\bar{x}=\frac{1}{N} \sum_{i=1}^N x_i,$

and the bootstrap sample mean $\bar{x}^\star_1$ is computed from a resampled dataset, the same applies for $\bar{x}^\star_2$. The sampling correlation between $\bar{x}^\star_1$ and $\bar{x}^\star_2$ is given by

$corr(\bar{x}^\star_1,\bar{x}^\star_2)= \frac{Cov(\bar{x}^\star_1,\bar{x}^\star_2)}{\sqrt{Var(\bar{x}^\star_1)Var(\bar{x}^\star_2)}},$

so, we need to begin with computing the covariance term $Cov(\bar{x}^\star_1,\bar{x}^\star_2)$ and the respective variance terms in order to calculate their correlation. We start by denoting that

$cov(\bar{x}_1^\star, \bar{x}_2^\star)=\frac{\sigma^2}{n}$

since $E[\bar{x}_1^\star]=E[\bar{x}_2^\star]$ which is the population mean. Now we want to calculate the variances.

However, this is where I get stuck. How do one calculate the variance of a bootstrap realization sample mean?

My first thought was that since the bootstrap sample is obtained by resampling with replacement, the variance of the sample mean for a bootstrap sample is the same as the variance of the original sample mean, i.e. $\sigma^2/n$, but this cannot be correct since this would not obtain $corr(\bar{x}^\star_1,\bar{x}^\star_2)=\frac{n}{2n-1} \approx 50\%$.

Any help would be appreciated in order to help me see what it is that I am missing here!

Update:

I (hopefully) solved it using the following definition

$Var(\bar{x}_i^\star)=\frac{1}{n^2}\left[ \sum_{i=1}^n Var(x_i) + 2\sum_{i=1}^{n-1}\sum_{j=i+1}^n Cov(x_i^\star,x_j^\star)\right] = \frac{1}{n^2}\left[n\sigma^2+n(n-1)\frac{\sigma^2}{n}\right]$

and obtained the variance $\frac{\sigma^2(2n-1)}{n^2}$. Note that $x_i^\star$ and $x_j^\star$ are resamples of the original dataset.

From this, I could insert the answers into the formula for correlation and prove that $corr(\bar{x}^\star_1,\bar{x}^\star_2)=\frac{n}{2n-1} \approx 50\%$. What do you think about this solution? Do you think it is correct (even though I did not use the bagged mean to prove it)?

$\endgroup$

1 Answer 1

2
$\begingroup$

I would do this by conditioning on the sample.

Write $s^2=\frac{1}{n}\sum_{i=1}^n (x_i-\bar x)^2$, so $s^2$ is the variance of the empirical distribution of the sample. Also, write $\mathrm{var}_P$ and $\mathrm{var}_B$ for the variance under sampling of the original $x$ and the variance under bootstrap resampling. All the bootstrap resamples share the same initial sample but have independent resampling.

We have

  • $\mathrm{var}_P[\bar x]=\sigma^2/n$
  • $\mathrm{var}_B[\bar x_1^*]=s^2/n$ (because it's just iid sampling from the empirical distribution)
  • $\mathrm{var}_B[\bar x_2^*]=s^2/n$ (ditto)

Now, the total variance of $\bar x_1^*$ is given by the conditional variance formula $$\mathrm{var}_{PB}[\bar x_1^*]=E_P[\mathrm{var}_B[\bar x_1^*]]+\mathrm{var}_P[E_B[\bar x_1^*]]=E_P[s^2/n]+\mathrm{var}_P[\bar x]=\frac{n-1}{n}\frac{s^2}{n}+\sigma^2/n$$

Two bootstrap means share the sampling components of the variance but have independent resampling components. $$\mathrm{cov}[\bar x_1^*,\bar x_2^*]= \sigma^2/n$$ so $$\mathrm{corr}[\bar x_1^*,\bar x_2^*]=\frac{\mathrm{cov}[\bar x_1^\star,\bar x_2^\star]}{\sqrt{\mathrm{var}[\bar x_1^\star]\mathrm{var}[\bar x_2^\star]}}=\frac{\sigma^2/n}{\frac{n-1}{n}\frac{s^2}{n}+\sigma^2/n}\approx 1/2$$

And if you bag $m$ times you reduce the resampling component by a factor of $m$, because they are all conditionally independent: $$\mathrm{var}_{PB}[\bar x_\textrm{bag}]=\sigma^2/n+\frac{1}{m}\frac{n-1}{n}\frac{s^2}{n}$$ which gets closer to $\sigma^2/n$ as $m$ increases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.