1
$\begingroup$

Let $\{X_1,X_2,\dots,X_n\}$ be a sample of $n$ iid observations of a random variable $X$, and let $\overline X_n = \frac{1}{n} \sum_{i=1}^n X_i$ be the sample mean.

Now suppose we want to use bootstrapping to estimate, for example, the variance of the sample mean. Let $\{X_1^{*(i)},X_2^{*(i)},\dots,X_n^{*(i)}\}$ be a sample with replacement from the original sample and let $\overline X_n^{*(i)} = \frac{1}{n} \sum_{j=1}^n X_j^{*(i)}$ be the sample mean for the $i$-th bootstrap sample.

Let $B$ be the number of bootstrap rounds. Then we have a bootstrap estimate of the variance. $$ \text{Var}_{B,n} = \frac{1}{B-1} \sum_{i=1}^B (\overline X_n^{*(i)} - \overline X_B^*)^2, \quad \quad \text{where} \ \overline X_B^* = \frac{1}{B} \sum_{i=1}^B \overline X_n^{*(i)}. $$

Suppose $n$ is very large and computing the bootstrap variance estimate will be computationally intensive. For example suppose we were using a slow computer and take $n=10^7$

To speed things up, we decide that instead of repeatedly drawing samples of size $n$ we will instead draw samples of size $m \ll n$ with replacement. For example, for $n=10^7$, we could take $m=5000$.

Let $\{Y_1^{*(i)},Y_2^{*(i)},\dots,Y_m^{*(i)}\}$ be a sample with replacement from the original sample $\{X_1,X_2,\dots,X_n\}$ and let $\overline Y_m^{*(i)} = \frac{1}{m} \sum_{j=1}^m Y_j^{*(i)}$ be the sample mean for the $i$-th bootstrap sample.

Now, this time we have the variance estimate: $$ \text{Var}_{B,m} = \frac{1}{B-1} \sum_{i=1}^B (\overline Y_m^{*(i)} - \overline Y_B^*)^2, \quad \quad \text{where} \ \overline Y_B^* = \frac{1}{B} \sum_{i=1}^B \overline Y_m^{*(i)}. $$

Is $\text{Var}_{B,m}$ a valid estimate of the variance of the sample mean? One issue that I have noticed is that since $m$ is such so much smaller than $n$ that we have effectively sampled without replacement when we computed $\text{Var}_{B,m}$. Is this a problem?

$\endgroup$
4
  • 1
    $\begingroup$ I must be missing something as$$\text{var}({\bar X}_n)=\frac{1}{n}\text{var}(X_i)$$and$$\text{var}{(\bar Y}_m)\approx\frac{1}{m}\text{var}(X_i)$$ $\endgroup$ – Xi'an Apr 5 at 14:06
  • $\begingroup$ "Now suppose we want to use bootstrapping to estimate, for example, the variance of the sample mean" Which variance do you wish to estimate? The variance of the sample $X_1,\dots,X_n$ or the variance of the population from which the sample $X_1,\dots,X_n$ is obtained? It makes no sense to use bootstrapping to estimate $\bar{X}$ and $\text{var}(X)$ when you can just as well use the sample $X_1,\dots,X_n$. The bootstrapping does not add any information. $\endgroup$ – Sextus Empiricus Apr 5 at 14:59
  • $\begingroup$ @SextusEmpiricus I am talking about estimating the variance of the sample mean, in analogy to estimating the variance of the sample median on the second page of these notes $\endgroup$ – ManUtdBloke Apr 5 at 15:52
  • $\begingroup$ @SextusEmpiricus I realise the bootstrap doesn't add any information, my post is a simple toy example to frame my question about what happens if we perform bootstrapping where we repeatedly draw samples of size $m$ with replacement from an original dataset of size $n$, in contrast to the standard bootstrap which repeatedly draws samples of size $n$ with replacement from the original dataset of size $n$. $\endgroup$ – ManUtdBloke Apr 5 at 15:54
2
$\begingroup$

I'm no expert in the bootstrap, but I anticipate that using fewer samples in your bootstrap means that the uncertainty in mean will be inflated.

The standard error of the sampling distribution is inversely proportional to the square root of the sample size. A smaller sample size thus means larger uncertainty in your estimates. Its easy to see evidence of this. Shown below are bootstrap samples of a dataset I've simulated. Blue histogram is resamples using the full size of the dataset. Red histogram is using 1% of the dataset.

Bootstrapping is embarrassingly parallelized. I'm willing to bet there is a good solution to this problem which does not require sub sampling of the data.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.