Say I have a sample and the bootstrap sample from this sample for a stastitic $\chi$ (e.g. the mean). As we all know, this bootstrap sample estimates the sampling distribution of the estimator of the statistic.

Now, is the mean of this bootstrap sample a better estimate of the population statistic than the statistic of the original sample? Under what conditions would that be the case?

  • 2
    $\begingroup$ The mean of the bootstrap sample is the mean of the sample and you do not need a bootstrap sample in this case. $\endgroup$ – Xi'an Jan 14 '15 at 14:58
  • 1
    $\begingroup$ Thanks @Xi'an I am not sure I follow. The mean of the bootstrap sample can be numerically different from the mean of the sample. Are you trying to say that the two are still theoretically equivalent? Can you confirm on both ends? $\endgroup$ – Amelio Vazquez-Reina Jan 14 '15 at 15:06
  • 2
    $\begingroup$ Let's get our terminology clear: "bootstrap sample" could refer either to a specific sample-with-replacement from the data or it could refer to a (multivariate) random variable of which such a sample would be considered one realization. You are correct that the mean of a realization can differ from the mean of the data, but @Xi'an provides the more relevant observation that the mean of the random variable (which by definition is the bootstrap estimate of the population mean) must coincide with the mean of the data. $\endgroup$ – whuber Jan 14 '15 at 15:55
  • 1
    $\begingroup$ Then your question is almost identical to stats.stackexchange.com/questions/126633/…; the only difference is that the bootstrap sample realizations can overlap, but the analysis given in the answer there is easily carried over to the bootstrap situation, with the same result. $\endgroup$ – whuber Jan 14 '15 at 16:46
  • 1
    $\begingroup$ I see the connection @whuber, although in bootstrap one has "subsets with replacement" and the realizations may overlap, as you said. I would imagine that the distribution (e.g. pseudorandomness) used to get the re-samples in bootstrap can also affect the bias of the estimate from the bootstrap sample. Perhaps the answer is that for all practical matters the difference is negligible. This is what the question is after: conditions, subtleties, and the difference in practice. $\endgroup$ – Amelio Vazquez-Reina Jan 14 '15 at 16:50

Let's generalize, so as to focus on the crux of the matter. I will spell out the tiniest details so as to leave no doubts. The analysis requires only the following:

  1. The arithmetic mean of a set of numbers $z_1, \ldots, z_m$ is defined to be

    $$\frac{1}{m}\left(z_1 + \cdots + z_m\right).$$

  2. Expectation is a linear operator. That is, when $Z_i, i=1,\ldots,m$ are random variables and $\alpha_i$ are numbers, then the expectation of a linear combination is the linear combination of the expectations,

    $$\mathbb{E}\left(\alpha_1 Z_1 + \cdots + \alpha_m Z_m\right) = \alpha_1 \mathbb{E}(Z_1) + \cdots + \alpha_m\mathbb{E}(Z_m).$$

Let $B$ be a sample $(B_1, \ldots, B_k)$ obtained from a dataset $x = (x_1, \ldots, x_n)$ by taking $k$ elements uniformly from $x$ with replacement. Let $m(B)$ be the arithmetic mean of $B$. This is a random variable. Then

$$\mathbb{E}(m(B)) = \mathbb{E}\left(\frac{1}{k}\left(B_1+\cdots+B_k\right)\right) = \frac{1}{k}\left(\mathbb{E}(B_1) + \cdots + \mathbb{E}(B_k)\right)$$

follows by linearity of expectation. Since the elements of $B$ are all obtained in the same fashion, they all have the same expectation, $b$ say:

$$\mathbb{E}(B_1) = \cdots = \mathbb{E}(B_k) = b.$$

This simplifies the foregoing to

$$\mathbb{E}(m(B)) = \frac{1}{k}\left(b + b + \cdots + b\right) = \frac{1}{k}\left(k b\right) = b.$$

By definition, the expectation is the probability-weighted sum of values. Since each value of $X$ is assumed to have an equal chance of $1/n$ of being selected,

$$\mathbb{E}(m(B)) = b = \mathbb{E}(B_1) = \frac{1}{n}x_1 + \cdots + \frac{1}{n}x_n = \frac{1}{n}\left(x_1 + \cdots + x_n\right) = \bar x,$$

the arithmetic mean of the data.

To answer the question, if one uses the data mean $\bar x$ to estimate the population mean, then the bootstrap mean (which is the case $k=n$) also equals $\bar x$, and therefore is identical as an estimator of the population mean.

For statistics that are not linear functions of the data, the same result does not necessarily hold. However, it would be wrong simply to substitute the bootstrap mean for the statistic's value on the data: that is not how bootstrapping works. Instead, by comparing the bootstrap mean to the data statistic we obtain information about the bias of the statistic. This can be used to adjust the original statistic to remove the bias. As such, the bias-corrected estimate thereby becomes an algebraic combination of the original statistic and the bootstrap mean. For more information, look up "BCa" (bias-corrected and accelerated bootstrap) and "ABC". Wikipedia provides some references.

  • $\begingroup$ You mean that the expectation of the bootstrap mean is equal to the data mean, no? The bootstrap mean itself is not determined by the (original) data sample. $\endgroup$ – capybaralet Jul 29 '15 at 0:06
  • 1
    $\begingroup$ @user2429920 The bootstrap mean is a statistic determined by the sample. In this sense it is identical to the sample mean. Its expectation is taken in the sense of the sampling distribution. I suspect you might be using "expectation" in a different sense relative to the process of computing the bootstrap mean via repeated subsampling with replacement. $\endgroup$ – whuber Jul 29 '15 at 1:09
  • 1
    $\begingroup$ I think the last paragraph is the actual answer to this question as it is general and not focused only on the mean statistic. I had the same doubt the OP did, and I wasn't aware of the existence of BCa. Although the demonstration in this answer did not help me much (I'm not using the mean as my statistic) the last paragraph was very clear about the crux of the matter. I believe Xi'an's answer also addresses the case where the mean statistic is used, so same issue. Thank you! $\endgroup$ – Gabriel Apr 19 '19 at 14:50
  • 1
    $\begingroup$ @Gabriel good points. I checked the record: before editing, this question originally asked only about the mean. That's why the answers appear to be so focused on that statistic. $\endgroup$ – whuber Apr 22 '19 at 22:05

Since the bootstrap distribution associated with an iid sample $X_1,\ldots,X_n$ is defined as$$\hat{F}_n(x) = \frac{1}{n}\sum_{i=1}^n\mathbb{I}_{X_i\le x}\qquad X_i\stackrel{\text{iid}}{\sim}F(x)\,,$$ the mean of the bootstrap distribution $\hat{F}_n$ (conditional on the iid sample $X_1,\ldots,X_n$) is$$\mathbb{E}_{\hat{F}_n}[X]=\frac{1}{n}\sum_{i=1}^n X_i=\bar{X}_ n$$ When you (if you have to) implement a simulation version of this expectation, i.e., compute an average of $B$ random draws from $\hat{F}_n$, $$\hat{\mathbb{E}}_{\hat{F}_n}[X]=\frac{1}{B} \sum_{b=1}^B X^*_b \qquad X^*_i\stackrel{\text{iid}}{\sim}\hat F_n(x)\,,$$there is some Monte Carlo variability in this approximation of $\mathbb{E}_{\hat{F}_n}[X]$, but its mean (the expectation of the empirical average, conditional on the original sample $X_1,\ldots,X_n$) and its limit when the number $B$ of bootstrap simulations grows to infinity are both exactly $\bar{X}_ n$.

  • 2
    $\begingroup$ +1 This is the answer I originally wanted to write, but feared it might be too opaque for some readers. I nevertheless am glad to see it so elegantly presented. I'm not sure what you mean in your last sentence, though, where you appear to differentiate the "expectation" of the simulated approximation to the mean from its "limit": since the expectation is constant (it does not vary with simulation size), there really isn't any limit to take. $\endgroup$ – whuber Jan 14 '15 at 17:15
  • $\begingroup$ @whuber: Thank you for the comment and sorry for writing my terse answer exactly at the same time as yours! Your explanations are certainly more readable by novices in bootstrap. I corrected the final sentence, whose limiting part is the law of large numbers. $\endgroup$ – Xi'an Jan 14 '15 at 17:24
  • 3
    $\begingroup$ Your use of "mean" in that last sentence is quite ambiguous! I figured it out from your LLN clue. For any finite simulation of the bootstrap distribution, each sample in the simulation produces its own mean (there's one meaning of "mean"). The average of all those samples in a given simulation produces a simulation mean (there's another meaning). The simulation mean converges to a constant as the simulation size grows large, which is the bootstrap mean (a third meaning), and this equals the sample mean (the fourth meaning). (And this estimates the population mean--a fifth meaning!) $\endgroup$ – whuber Jan 14 '15 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.