# What is the fundamental "problem" why bootstrap intervals tend to be too short?

I found several posts which perform simulations and demonstrate that bootstrap intervals tend to be too short (even when accounting for the correct dependency/grouping structure). This is repeatedly asked and described in answers which point to the fact that bootstrap is only asymptotically valid.

E.g. this post and the discussions link a lot of questions Is bootstrap problematic in small samples?

What I did not find is a discussion about the why.

Is the fundamental reason the following: By exchanging sampling from an infinite population with sampling from a finite population we are

1. introducing artificial correlation between resamples (drawing with replacement) which was previously not there (since the samples were initially drawn at random)
2. we have exchanged an infinite population with a finite population so that we have a similar effect like for a finite population $$\operatorname{Var}\left( \frac1n \sum_i X_i \right) =\frac{1}{n}(1-\frac{n}{N}) \sigma^2$$ which would be zero for n=N. So that the finite population (the sample from which we draw the resamples) just behaves differently from the true infinite population. But I cannot fully wrap my head around this. Maybe someone sees a connection to the altered finite population statistic which I quoted above...
3. ....

Or is it a mix of both 1) and 2)?

Maybe discussing this why question with the min/max statistics would help? (I know common work arounds like $$n$$ out of $$m$$ bootstrap and I am not looking for adaptations of the vanilla bootstrap but for an answer WHY the vanilla bootstrap fails in certain situations).

• Data-driven means what it says: Your estimates of variability are based realistically on what the data -- meaning literally, observed data -- are, not on some more or less hypothetical or fantastic or Utopian set of assumptions. But, what bites too: the bootstrap can't know about what might have been observed but never was. At least, that is my attempt at a grade school version. Commented May 7 at 21:23
• “bootstrap intervals tend to be too short“, certainly they can be, just like any other interval. Could you be more specific about what you mean with too short. Too short for what? Commented May 8 at 9:27
• The link that you provided does not discuss bootstrap intervals being too small, but bootstrap intervals being imprecise. Commented May 8 at 9:28
• Possibly relevant: Do the 2.5th and 97.5th percentile of the theoretical sampling distribution of a statistic always contain the true population parameter? and the answer by Ute, "Bootstrap percentile intervals are not frequentist confidence intervals". Bootstrap intervals are not only short, they can also be wrong. Like a bootstrap interval for estimating a maximum will never contain the true maximum. Commented May 8 at 9:41
• In this question a case of small confidence intervals is discussed On the high dimensional bootstrap (but at the same time there are situations of too large intervals in that question). In my opinion it is only small because it is computed badly, and not because the bootstrap is fundamentally wrong. If you sample residuals instead of the true errors, then you have a different distribution with smaller variance and you need to correct for that. In a same way an uncorrected sample standard deviation is biased. Commented May 8 at 9:48

A correct confidence interval (CI) at level 95%, say, for a parameter defined within a probability model has the property that if indeed observations are generated from that model, with any parameter value, the probability that the CI will catch the true parameter is 95%.

In order to prove theoretically that a CI is correct, we therefore need to assume that we know that data were generated from that model.

In nonparametric bootstrapping, we pretend that the data are generated from the empirical distribution of the data instead. This ignores the additional uncertainty that comes into the procedure from not knowing how well the data in fact represent the true underlying distribution. We pretend that it does, but (particularly with small samples) this may not necessarily be very good. Bootstrap CIs can be systematically too short because CIs are meant to capture the uncertainty in the data regarding the parameter estimator, but the bootstrap CI only accounts for the variation visible in the data, but not for potential additional variation between the empirical distribution of the data and the underlying true data generating process. (In many situations this vanishes for $$n\to\infty$$.)

(If I understand your points 1 and 2 well, it's neither of these.)

• Another way to say it is that the bootstrap attempts to simulate the true repeated independent sampling distribution from an estimator with the distribution from repeated samples with replacement from one dataset. In many cases the bootstrap distribution does not resemble the sampling distribution, and confidence intervals from the bootstrap will be inaccurate in at least one of the tails. Commented May 7 at 12:06
• Thank you for your clarification, for a second I forgot to see how sampling with replacement ensures the independent sample property. Commented May 7 at 19:39

There is nothing to really add to Christian Hennig's excellent answer.

However, I will reverse your question. Should one not really wonder if parametric CI's are too wide?

If one were to truly sample from say $$N(0,1)$$, as we keep drawing more and more random samples, one would expect to see (with very low, but non-zero probability) large negative and positive values (i.e. values beyond $$~2\sigma$$). The parametric CI's account for this, hence they are larger than bootstrapped CI's. The bootstrapped CI's are fundamentally censored: the max or min value they can encounter are limited by the original sample.

While the parametric samples are "correct" from a mathematical statistics pov, from an applied statistics pov, we are in fact dealing with censored data. No matter what we are measuring (dimensions, voltages, blood glucose levels, etc.), this data is censored: it is censored by the nature of what we measure (there are no negative glucose levels, or shaft lengths), they are censored by the range of our measuring instrument, and by the practical range of what we measure (the lengths of the shafts I measure practically have a very narrow range, possibly less than $$+/-2\sigma$$). So, in applied statistics, the probability of observing very large or very small values is in fact 0. So we could ask whether the parametric CI's may be too wide?

If I were to really measure shaft lengths, produced by a process incapable of units longer or shorter than some limits (limits which I may not know with certainty, but quite narrow limits), I would tend to trust a boostrapped CI more than a parametric one, provided that my sample was large enough to be representative. In this specific case, a sample size of say 20 or 30 should be adequate. In other cases, ?

Yes, bootstrap on small samples will be problematic (there is no free lunch; if we can dispense from distributional assumptions, we will have to pay with sample size). How small is too small? I will leave this for others to debate, but the usual answer should apply: it depends.... My only point is that, in applied statistics, where we deal with censored data, a bootstrap from a "representative" sample may be closer to the truth than a parametric one. Food for thought...