Assumptions regarding bootstrap estimates of uncertainty

Question

I appreciate the usefulness of the bootstrap in obtaining uncertainty estimates, but one thing that's always bothered me about it is that the distribution corresponding to those estimates is the distribution defined by the sample. In general, it seems like a bad idea to believe that our sample frequencies look exactly like the underlying distribution, so why is it sound/acceptable to derive uncertainty estimates based on a distribution where the sample frequencies define the underlying distribution?

On the other hand, this may be no worse (possibly better) than other distributional assumptions we typically make, but I'd still like to understand the justification a bit better.

Answer 1

There are several ways that one can conceivably apply the bootstrap. The two most basic approaches are what are deemed the "nonparametric" and "parametric" bootstrap. The second one assumes that the model you're using is (essentially) correct.

Let's focus on the first one. We'll assume that you have a random sample $X_1, X_2, \ldots, X_n$ distributed according the the distribution function $F$. (Assuming otherwise requires modified approaches.) Let $\hat{F}_n(x) = n^{-1} \sum_{i=1}^n \mathbf{1}(X_i \leq x)$ be the empirical cumulative distribution function. Much of the motivation for the bootstrap comes from a couple of facts.

Dvoretzky–Kiefer–Wolfowitz inequality

$$ \renewcommand{\Pr}{\mathbb{P}} \Pr\big( \textstyle\sup_{x \in \mathbb{R}} \,|\hat{F}_n(x) - F(x)| > \varepsilon \big) \leq 2 e^{-2n \varepsilon^2} \> . $$

What this shows is that the empirical distribution function converges uniformly to the true distribution function exponentially fast in probability. Indeed, this inequality coupled with the Borel–Cantelli lemma shows immediately that $\sup_{x \in \mathbb{R}} \,|\hat{F}_n(x) - F(x)| \to 0$ almost surely.

There are no additional conditions on the form of $F$ in order to guarantee this convergence.

Heuristically, then, if we are interested in some functional $T(F)$ of the distribution function that is smooth, then we expect $T(\hat{F}_n)$ to be close to $T(F)$.

(Pointwise) Unbiasedness of $\hat{F}_n(x)$

By simple linearity of expectation and the definition of $\hat{F}_n(x)$, for each $x \in \mathbb{R}$,

$$ \newcommand{\e}{\mathbb{E}} \e_F \hat{F}_n(x) = F(x) \>. $$

Suppose we are interested in the mean $\mu = T(F)$. Then the unbiasedness of the empirical measure extends to the unbiasedness of linear functionals of the empirical measure. So, $$ \e_F T(\hat{F}_n) = \e_F \bar{X}_n = \mu = T(F) \> . $$

So $T(\hat{F}_n)$ is correct on average and since $\hat{F_n}$ is rapidly approaching $F$, then (heuristically), $T(\hat{F}_n)$ rapidly approaches $T(F)$.

To construct a confidence interval (which is, essentially, what the bootstrap is all about), we can use the central limit theorem, the consistency of empirical quantiles and the delta method as tools to move from simple linear functionals to more complicated statistics of interest.

Good references are

1. B. Efron, Bootstrap methods: Another look at the jackknife, Ann. Stat., vol. 7, no. 1, 1–26.
2. B. Efron and R. Tibshirani, An Introduction to the Bootstrap, Chapman–Hall, 1994.
3. G. A. Young and R. L. Smith, Essentials of Statistical Inference, Cambridge University Press, 2005, Chapter 11.
4. A. W. van der Vaart, Asymptotic Statistics, Cambridge University Press, 1998, Chapter 23.
5. P. Bickel and D. Freedman, Some asymptotic theory for the bootstrap. Ann. Stat., vol. 9, no. 6 (1981), 1196–1217.

Answer 2

Here is a different approach to thinking about it:

Start with the theory where we know the true distribution, we can discover properties of sample statistics by simulating from the true distribution. This is how Gosset developed the t-distribution and t-test, by sampling from known normals and computing the statistic. This is actually a form of the parametric bootstrap. Note that we are simulating to discover the behavior of the statistics (sometimes relative to the parameters).

Now, what if we do not know the population distribution, we have an estimate of the distribution in the empirical distribution and we can sample from that. By sampling from the empirical distribution (which is known) we can see the relationship between the bootstrap samples and the empirical distribution (the population for the bootstrap sample). Now we infer that the relationship from bootstrap samples to empirical distribution is the same as from the sample to the unknown population. Of course how well this relationship translates will depend on how representative the sample is of the population.

Remember that we are not using the means of the bootstrap samples to estimate the population mean, we use the sample mean for that (or whatever the statistic of interest is). But we are using the bootstrap samples to estimate properties (spread, bias) of the sampling proccess. And using sampling from a know population (that we hope is representative of the population of interest) to learn the effects of sampling makes sense and is much less circular.

Answer 3

The main trick (and sting) of bootstrapping is that it is an asymptotic theory: if you have an infinite sample to start with, the empirical distribution is going to be so close to the actual distribution that the difference is negligible.

Unfortunately, bootstrapping is often applied in small sample sizes. The common feel is that bootstrapping has shown itself to work in some very non-asymptotic situations, but be careful nonetheless. If your samplesize is too small, you are in fact working conditionally on your sample being a 'good representation' of the true distribution, which leads very easily to reasoning in circles :-)

Answer 4

I would argue not from the perspective of "asymptotically, the empirical distribution will be close to the actual distribution" (which, of course, is very true), but from a "long run perspective". In other words, in any particular case, the empirical distribution derived by bootstrapping will be off (sometimes shifted too far this way, sometimes shifted too far that way, sometimes too skewed this way, sometimes too skewed that way), but on average it will be a good approximation to the actual distribution. Similarly, your uncertainty estimates derived from the bootstrap distribution will be off in any particular case, but again, on average, they will be (approximately) right.