Bootstrapping won't always return population statistics - so why say it does?

Question

I've prowled the interwebs and looked at different questions and answers on the site, including this one: here. But nothing that I've found addresses the following problem.

Suppose you flip a fair coin 10 times and receive all heads (let heads be equal to 1 and tails equal to 0). Thus, the sample mean and variance statistic is 1 and 0 respectively. Furthermore, bootstrapping will always return a mean and variance of 1 and 0 respectively as well. Now, there is just under 0.1% chance of flipping 10 heads in a row, assuming a fair coin... but what if you have that 1 in 1000 sample? Bootstrapping is only going to return your sample statistic....

I am asking this question because I've heard it loosely stated that bootstrapping returns the population statistic.

EDIT: The other reason I am asking this question is to point out that bootstrapping has its limitations and cannot prevent against bad samples (I admit that the example is rather contrived). As @whuber pointed out, bootstrapping is an asymptotic statistic. However, is there any way that we can decide how close our bootstrap statistic is to the true population statistic (or derive some sort of "confidence" measure - I'm thinking about concentration inequalities here)?

Answer 1

Heuristically, you can think that the motivation behind the bootstrap is that given a large sample, your sample should be distributed approximately equal to your population. If your sample is distributed approximately equal to your population, then re-sampling from your sample and calculating your statistic should be approximately the same as re-sampling from your population and calculating your statistic. Of course, it will not be exactly the same, as your sample is not exactly distributed the same as your population.

So in your example, the issue is that you have drawn a sample that looks very different than your population, and thus your statistics calculated from resampling from your sample will look very different than statistics calculated from resampling from your population.

Despite your example, bootstrap is still a valid procedure because for large samples, large deviations in the distribution of your sample vs. the distribution of your population become less and less likely.

To say that "bootstrapping returns the population statistic" does not make sense.

Answer 2

First, bootstrap cannot remedy a problem of an unrepresentative original sample. Thus I agree with @CliffAB.

The chances of randomly drawing a sample that is little representative diminish as the sample size grows. If there was just one coin throw, the outcome would always be "crippled": one element (either head or tail) of the two (head, tail) that are in the population would be completely absent in the sample. Meanwhile, as @justanotherbrain notes in the question, the chances of having 10 heads out of 10 coin throws are just below $10^{-3}$ -- much better than when you have only one throw. @whuber correctly notes that bootstrap properties are asymptotic. I will add that, under certain conditions, bootstrap is more efficient than standard estimators, e.g. achieving convergence rate of $n^{3/2}$ rather than $n^{1/2}$. (I will not expand on these conditions here; a detailed treatment is given in Hall "The Bootstrap and Edgeworth Expansion" (1992) (this is a whole book).)

But let me provide another perspective why bootstrap is generally relevant. Sorry if it does not address your question directly enough.

Bootstrap is useful for assessing whether an estimator (a formula) works alright. The idea is as follows.

1. Consider the original sample (of $n$ elements) as "a population"
2. Draw an $n$-element bootstrap sample from the original sample
3. Apply the estimator (the formula) onto the bootstrap sample to obtain its realization (a value)
4. Iterate 2. and 3. many times, save the realizations of each iteration

The realizations from the many iterations will form an empirical distribution. Its characteristics (mean, variance, whatever else) will be accessible to you.

Since you consider the original sample as "a population", you know the true characteristics of that population. You can then take at the empirical distribution (based on the many bootstrap realization) from point 5. and see how it looks relative to the actual population characteristic (one item) that the estimator was supposed to estimate.

You can thus get an idea whether your estimator (a formula) is biased, how large its variance is etc. All that is valuable information when it comes to assessing properties of an estimator. Even better, the original sample should normally be fairly representative (contrary to your example) of the actual population of interest. Hence, you can extend your conclusion from the bootstrap experiment to the original problem of interest (by analogy).

For example, if you see that the estimator tends to underestimate the population characteristic by $b$ units ($b$ stands for bias), you would add $b$ to the estimate based on the original sample so as to remove the bias. That way you would get an unbiased estimate of the characteristic that you were after in the first place. Thus you would fruitfully utilize what you have learned about the estimator when bootstrapping.

But at the end I have to repeat that bootstrap is not a remedy in case of an unrepresentative original sample, as your example illustrates.