Explaining to laypeople why bootstrapping works

I recently used bootstrapping to estimate confidence intervals for a project. Someone who doesn't know much about statistics recently asked me to explain why bootstrapping works, i.e., why is it that resampling the same sample over and over gives good results. I realized that although I'd spent a lot of time understanding how to use it, I don't really understand why bootstrapping works.

Specifically: if we are resampling from our sample, how is it that we are learning something about the population rather than only about the sample? There seems to be a leap there which is somewhat counter-intuitive.

I have found a few answers to this question here which I half-understand. Particularly this one. I am a "consumer" of statistics, not a statistician, and I work with people who know much less about statistics than I do. So, can someone explain, with a minimum of references to theorems, etc., the basic reasoning behind the bootstrap? That is, if you had to explain it to your neighbor, what would you say?

-
(+1) You might mention briefly the questions you have looked at, but that don't quite satisfy you. There are lots of questions on the bootstrap here. :) –  cardinal Apr 8 '12 at 21:11
@cardinal Thanks, I updated the original post. Hopefully it is more clear. :) –  Alan H. Apr 9 '12 at 3:17

fwiw the medium length version I usually give goes like this:

Imagine you decide to make assumptions, e.g. that it is Normal, or Bernoulli or some other convenient fiction. Following the previous strategy you could again learn about how much the answer to your question when asked of a sample might vary depending on which particular sample you happened to get by repeatedly generating samples of the same size as the one you have and asking them the same question. That would be straightforward to the extent that you chose computationally convenient assumptions. (Indeed particularly convenient assumptions plus non-trivial math may allow you to bypass the sampling part altogether, but we will deliberately ignore that here.)

This seems like a good idea provided you are happy to make the assumptions. Imagine you are not. An alternative is to take the sample you have and sample from it instead. You can do this because the sample you have is also a population, just a very small discrete one; it looks like the histogram of your data. Sampling 'with replacement' is just a convenient way to treat the sample like it's a population and to sample from it in a way that reflects its shape.

This is a reasonable thing to do because not only is the sample you have the best, indeed the only information you have about what the population actually looks like, but also because most samples will, if they're randomly chosen, look quite like the population they came from. Consequently it is likely that yours does too.

For intuition it is important to think about how you could learn about variability by aggregating sampled information that is generated in various ways and on various assumptions. Completely ignoring the possibility of closed form mathematical solutions is important to get clear about this.

-
Nice answer. I especially like the penultimate paragraph. –  Peter Flom Apr 8 '12 at 23:02
(+1) This is a good answer. I think there may be a way to further draw out a very important point, though. In the way the bootstrap is normally carried out, there are two effects that are happening. First, we are pretending that the sample we have obtained is a proxy for our population. This is nominally a reasonable thing to do, provided our sample size is reasonably large. However, we usually have a hard time calculating the actual quantities of interest from that pretend distribution. So, we have to estimate them, and this is why we draw lots of bootstrap samples. If we could.../... –  cardinal Apr 9 '12 at 0:29
.../...calculate the quantities of interest directly for our pretend distribution, we'd prefer to do that. And, that would be the real bootstrap. But, usually we can't, so we're reduced to having to resampling, instead. –  cardinal Apr 9 '12 at 0:32
@naught101: "Reasonably large" can be quantified pretty well by the D-K-W inequality (if you'd like, you can look at my answer in the link in the OP's question) and regarding lots, it depends on the sample statistic of interest, but if we have $B$ bootstrap samples, then with simple Monte Carlo we know that the standard error is of order roughly $O(B^{-1/2})$. –  cardinal Apr 9 '12 at 2:11
@cardinal: Nice comment. A lot of people think that the bootstrap and resampling are the same thing when in fact the latter is a tool used for the former. A similar misconception is that many users of statistics tend to get MCMC and Bayesian analysis confused. –  MånsT Apr 10 '12 at 7:34

+1 to @ConjugatePrior, I just want to bring out one point which is implicit in his answer. The question asks, "if we are resampling from our sample, how is it that we are learning something about the population rather than only about the sample?" Resampling is not done to provide an estimate of the population distribution--we take our sample itself as a model of the population. Rather, resampling is done to provide an estimate of the sampling distribution of the sample statistic in question.

-
(+1) This is close to the point I was trying to make in the comment to ConjugatePrior's answer, though you've stated it more concisely and clearly. In some special cases, we can calculate the sampling distribution of the test statistic exactly under the empirical distribution obtained from the sample. But, usually, we can't and so we're forced into simulation. :) –  cardinal Apr 9 '12 at 1:13
I see, so if I understand you, then this technique assumes that the sample is an adequate model of the population, and therefore that resampling over that sample on a large enough scale will reveal something about the population, but only to the extent that the original sample is a good one. Now that I put it that way it seems almost obvious... –  Alan H. Apr 9 '12 at 1:31
@AlanH., I just want to change "... will reveal something about the population" to "... will reveal something about the sampling distribution" (of the statistic at issue, eg mean). But, yes, you have it there –  gung Apr 9 '12 at 1:52
I'd love to know the reason behind the downvote. If there's something wrong with this answer, perhaps I could fix it, it I knew what it was. –  gung Jun 25 '12 at 4:10
You're all correct, of course. Personally, and purely for pedagogical reasons, I save this point for my 'longer version', because in my particular audiences this point tends to knock their young and still unsteady intuitions a bit off balance if applied too soon. –  conjugateprior Jun 25 '12 at 12:37
show 1 more comment

Here are some animations which may help: http://www.stat.auckland.ac.nz/~wild/BootAnim/

-
+1, those are cool! –  gung Apr 10 '12 at 21:38
+1 - Very nice illustration! –  Max Gordon Apr 11 '12 at 6:29
Yes they helped me understand the intuition a lot better. Credits to my teacher for providing me with the link though. Nice that you guys like them! –  C. Pieters Apr 11 '12 at 12:57

This is probably a more technical explanation aimed at people who understand some statistics and mathematics (calculus, at least). Here's a slide from a course on survey bootstraps that I taught some while ago:

Some explanations are needed, of course. $T$ is the procedure to obtain the statistic from the existing data (or, to be technically precise, a functional from the distribution function to real numbers; e.g., the mean is $E[X]=\int x {\rm d}F$, where for the sample distribution function $F_n()$, the ${\rm d}F$ is understood as a point mass at a sample point). In the population, denoted by $F()$, application of $T$ gives the parameter of interest $\theta$. Now, we've taken a sample (the first arrow on the top), and have the empirical distribution function $F_n()$ -- we apply $T$ to it to obtain the estimate $\hat\theta_n$. How far is it from $\theta$, we wonder? What is the distribution that the random quantity $\hat\theta_n$ may have around $\theta$? This is the question mark in the lower left of the diagram, and this is the question the bootstrap tries to answer. To restate gung's point, this is not the question about the population, but the question about a particular statistic and its distribution.

If we could repeat our sampling procedure, we could get that distribution and learn more. Well, that usually is beyond our capabilities. However, if

1. $F_n$ is close enough to $F$, in a suitable sense, and
2. the mapping $T$ is smooth enough, i.e., if we take small deviations from $F()$, the results will be mapped to numbers close to $\theta$,

we can hope that the bootstrap procedure will work. Namely, we pretend that our distribution is $F_n()$ rather than $F()$, and with that we can entertain all possible samples -- and there will be $n^n$ such samples, which is only practical for $n\le 5$. Let me repeat again: the bootstrap works to create the sampling distribution of $\hat\theta_n^*$ around the "true" parameter $\hat\theta_n$, and we hope that with the two above conditions, this sampling distribution is informative about the sampling distribution of $\hat\theta_n$ around $\theta$:

$$\hat\theta_n^* \mbox{ to } \hat\theta_n \mbox{ is like } \hat\theta_n \mbox{ to } \theta$$

Now, instead of just going one way along the arrows, and losing some information/accuracy along these arrows, we can go back and say something about variability of $\hat\theta_n^*$ around $\hat\theta_n$.

The above conditions are spelled out it utmost technicality in Hall's (1991) book. The understanding of calculus that I said may be required as a prerequisite to staring at this slide is the second assumption concerning smoothness: in more formal language, the functional $T$ must possess a weak derivative. The first condition is, of course, an asymptotic statement: the larger your sample, the closer $F_n$ should become to $F$; and the distances from $\hat\theta_n^*$ to $\hat \theta_n$ should be the same order of magnitude as those from $\hat\theta_n$ to $\theta$. These conditions may break, and they do break in a number of practical situations with weird enough statistics and/or sampling schemes that do not produce empirical distributions that are close enough to $F$.

Now, where does that 1000 samples, or whatever the magic number might be, comes from? It comes from our inability to draw all $n^n$ samples, so we just take a random subset of these. The right most "simulate" arrow states another approximation that we are making on our way to get the distribution of $\hat\theta_n$ around $\theta$, and that is to say that our Monte Carlo simulated distribution of $\hat\theta_n^{(*r)}$ is a good enough approximation of the complete bootstrap distribution of $\hat\theta_n^*$ around $\hat\theta_n$.

-

I am answering this question because I agree that this is a difficult thing to do and there are many misconceptions. Efron and Diaconis attempted to do that in their 1983 Scientific American article and in my view they failed. There are several books out now devoted to the bootstrap that do a good job. Efron and Tibshirani do a great job in their article in Statistical Science in 1986. I tried especially hard to make bootstrap accessible to practitioner's in my bootstrap methods book and my introdcution to bootstrap with applications to R. Hall's book is great but very advanced and theoretical. Tim Hesterberg has written a great supplemental chapter to one of David Moore's introductory statistics books. The late Clifford Lunneborg had a nice book. Chihara and Hesterberg recently came out with an intermediate level mathematical statistics book that covers the bootstrap and other resampling methods. Even advanced books like Lahiri's or Shao and Tu's give good conceptual explanations. Manly does well with his book that covers permutations and the bootstrap There is no reason to be puzzled about the bootstrap anymore. It is important to keep in mind that the bootstrap depends on the bootstrap principle "Sampling with replacement behaves on the original sample the way the original sample behaves on a population. There are examples where this principle fails. It is important to know that the bootstrap is not the answer to every statistical problem.

Here are amazon links to all the books I mentioned and more.

Mathematical Statistics with Resampling and R

Bootstrap Methods and their Application

Bootstrap Methods: A Guide for Practitioners and Researchers

An Introduction to Bootstrap Methods with Applications to R

Resampling Methods for Dependent Data

Randomization, Bootstrap and Monte Carlo Methods in Biology

An Introduction to the Bootstrap

The Practice of Business Statistics Companion Chapter 18: Bootstrap Methods and Permutation Tests

Data Analysis by Resampling: Concepts and Applications

The Jackknife, the Bootstrap, and Other Resampling Plans

The Jackknife and Bootstrap

Permutation, Parametric, and Bootstrap Tests of Hypotheses

The Bootstrap and Edgeworth Expansion

-
@Proc: Thanks for those edits. That looks nicer. :) –  cardinal Jul 24 '12 at 18:42
I have formated the answer to make it readable. I urge you, in the most emphatic manner and with all the due respect, to format your answers in order to improve readability. I know that this is a tortuous road but people would be very benefitted from this and I am sure that upvotes would rain on your answers. –  user10525 Jul 24 '12 at 18:49
@Procrastinator. I am doing that more often. in some cases I am in a hurry to get my answer posted and come back to clean it up later. I haven't got the hang of converting link addresses to links by title and I am not sure that it is all that necessary. It is a single click either way But if you can't wait for that I don't mind you doing the edits. In fact I appreciate it. –  Michael Chernick Jul 24 '12 at 18:56
I was going to change my comment to "I don't mind you doing the edits" with the "But if you can't wait" taken out. I see how what you did is neater and easier and probably takes less time but I just haven't learned it yet and I don't see this as such a big deal the way some moderators and other members do. –  Michael Chernick Jul 24 '12 at 19:05
(+1) I confer on you the power of the $10,000$ points @Michael Chernick. –  user10525 Jul 24 '12 at 19:11
show 1 more comment

Through bootstrapping you are simply taking samples over and over again from the same group of data (your sample data) to estimate how accurate your estimates about the entire population (what really is out there in the real world) is.

If you were to take one sample and make estimates on the real population, you might not be able to estimate how accurate your estimates are - we only have one estimate and have not identified how this estimate varies with different samples that we might have encountered.

With bootstrapping, we use this main sample to generate multiple samples. For example, if we measured the profit every day over 1000 days we might take random samples from this set. We might the profit from one random day, record it, get the profit from another random day (which might happen to be the same day as before - sampling with replacement), record it, and so forth, until we get a "new" sample of 1000days (from the original sample).

This "new" sample is not identical to the original sample - indeed we might generate several "new" samples as above. When we look at the variations in the means and estimate, we are able to get a reading on how accurate the original estimates were.

Edit - in response to comment

The "newer" samples are not identical to the first one and the new estimates based on these will vary. This simulates repeated samples of the population. The variations in the estimates of the "newer" samples generated by the bootstrap will shed a light on how the sample estimates would vary given different samples from the population. This is in fact how we can get try to measure the accuracy of the original estimates.

Of course, instead of bootstrapping you might instead take several new samples from the population but this might be infeasible.

-
Thanks! This much I understand. I am particularly wondering how it is that resampling from a sample of the population helps to understand the underlying population. If we are resampling from a sample, how is it that we are learning something about the population rather than only about the sample? There seems to be a leap there which is somewhat counter-intuitive. –  Alan H. Apr 8 '12 at 21:26