# Bootstrapping won't always return population statistics - so why say it does?

I've prowled the interwebs and looked at different questions and answers on the site, including this one: here. But nothing that I've found addresses the following problem.

Suppose you flip a fair coin 10 times and receive all heads (let heads be equal to 1 and tails equal to 0). Thus, the sample mean and variance statistic is 1 and 0 respectively. Furthermore, bootstrapping will always return a mean and variance of 1 and 0 respectively as well. Now, there is just under 0.1% chance of flipping 10 heads in a row, assuming a fair coin... but what if you have that 1 in 1000 sample? Bootstrapping is only going to return your sample statistic....

I am asking this question because I've heard it loosely stated that bootstrapping returns the population statistic.

EDIT: The other reason I am asking this question is to point out that bootstrapping has its limitations and cannot prevent against bad samples (I admit that the example is rather contrived). As @whuber pointed out, bootstrapping is an asymptotic statistic. However, is there any way that we can decide how close our bootstrap statistic is to the true population statistic (or derive some sort of "confidence" measure - I'm thinking about concentration inequalities here)?

Heuristically, you can think that the motivation behind the bootstrap is that given a large sample, your sample should be distributed approximately equal to your population. If your sample is distributed approximately equal to your population, then re-sampling from your sample and calculating your statistic should be approximately the same as re-sampling from your population and calculating your statistic. Of course, it will not be exactly the same, as your sample is not exactly distributed the same as your population.

So in your example, the issue is that you have drawn a sample that looks very different than your population, and thus your statistics calculated from resampling from your sample will look very different than statistics calculated from resampling from your population.

Despite your example, bootstrap is still a valid procedure because for large samples, large deviations in the distribution of your sample vs. the distribution of your population become less and less likely.

To say that "bootstrapping returns the population statistic" does not make sense.

• I agree with this answer - it is concise and clears up a misconception that I've heard many times. Believe it or not, I've talked to a few scientists that believe bootstrapping returns the population statistic (which, as you said, doesn't make sense). I admit that my example is contrived - but is for illustration purposes. In the case of only 50 coin flips, for example, the chances of flipping all heads is only about 8.8^-14%. So the key point is that bootstrapping is good for large samples, in fact some authors recommend no less than 50 sample. – justanotherbrain Jun 19 '15 at 16:53

First, bootstrap cannot remedy a problem of an unrepresentative original sample. Thus I agree with @CliffAB.

The chances of randomly drawing a sample that is little representative diminish as the sample size grows. If there was just one coin throw, the outcome would always be "crippled": one element (either head or tail) of the two (head, tail) that are in the population would be completely absent in the sample. Meanwhile, as @justanotherbrain notes in the question, the chances of having 10 heads out of 10 coin throws are just below $10^{-3}$ -- much better than when you have only one throw. @whuber correctly notes that bootstrap properties are asymptotic. I will add that, under certain conditions, bootstrap is more efficient than standard estimators, e.g. achieving convergence rate of $n^{3/2}$ rather than $n^{1/2}$. (I will not expand on these conditions here; a detailed treatment is given in Hall "The Bootstrap and Edgeworth Expansion" (1992) (this is a whole book).)

But let me provide another perspective why bootstrap is generally relevant. Sorry if it does not address your question directly enough.

Bootstrap is useful for assessing whether an estimator (a formula) works alright. The idea is as follows.

1. Consider the original sample (of $n$ elements) as "a population"
2. Draw an $n$-element bootstrap sample from the original sample
3. Apply the estimator (the formula) onto the bootstrap sample to obtain its realization (a value)
4. Iterate 2. and 3. many times, save the realizations of each iteration

The realizations from the many iterations will form an empirical distribution. Its characteristics (mean, variance, whatever else) will be accessible to you.

Since you consider the original sample as "a population", you know the true characteristics of that population. You can then take at the empirical distribution (based on the many bootstrap realization) from point 5. and see how it looks relative to the actual population characteristic (one item) that the estimator was supposed to estimate.

You can thus get an idea whether your estimator (a formula) is biased, how large its variance is etc. All that is valuable information when it comes to assessing properties of an estimator. Even better, the original sample should normally be fairly representative (contrary to your example) of the actual population of interest. Hence, you can extend your conclusion from the bootstrap experiment to the original problem of interest (by analogy).

For example, if you see that the estimator tends to underestimate the population characteristic by $b$ units ($b$ stands for bias), you would add $b$ to the estimate based on the original sample so as to remove the bias. That way you would get an unbiased estimate of the characteristic that you were after in the first place. Thus you would fruitfully utilize what you have learned about the estimator when bootstrapping.

But at the end I have to repeat that bootstrap is not a remedy in case of an unrepresentative original sample, as your example illustrates.

• All this is very good information. After reading through I was still unsure that it addresses the current question, though. The question seems to ask for ways to identify when the bootstrap might fail and why. Although you touch on that with a discussion of representativeness, how can we tell when a sample is not representative? Under what conditions could we have some assurance that bootstrap procedures are giving reliable results? – whuber Jun 19 '15 at 12:50
• @whuber, Yes, I sense that my answer is too remote. At the same time, I am afraid I am not competent enough to address the points that you just raised. Should I leave the answer as is or delete it, if there is little hope that I would improve it any time soon? – Richard Hardy Jun 19 '15 at 12:55
• As always, it's your call. I genuinely think everything in this post is valid, useful, and clearly explained. At the least, we might hope that by leaving this post up it could help the O.P. formulate the question more precisely. I also think that it would take only a little more work for you to address the question, because you have laid a good foundation. Doesn't it come down to the fact that the bootstrap's properties (like those of ML) are asymptotic? Thus, with small samples we have little right to expect the bootstrap always to perform well--and the O.P.'s example reveals why not. – whuber Jun 19 '15 at 13:00
• I like this answer - it doesn't directly address the current question but it is concise - I might borrow it to answer a few questions in the future, so please keep it. Also, @whuber, great point about bootstrap being asymptotic. Do either of you know of using a concentration inequality or something similar to give some sort of "confidence" on the bootstrap statistic? I'm intentionally being vague - I haven't heard of anything like it before. – justanotherbrain Jun 19 '15 at 17:01