Skip to main content
3 of 11
added 198 characters in body
EdM
  • 101.5k
  • 11
  • 102
  • 303

The main problem is that bootstrapping to determine anything close to extreme values of a distribution, as needed for the percentile bootstrap, is unreliable. This issue is discussed elsewhere on this site, for example here and here. Essentially, the few values available at the tails of the empirical distribution for any particular sample might not represent the underlying distribution very well. An extreme but illustrative case is trying to use bootstrapping to estimate the maximum order statistic of a random sample from a uniform $\;\mathcal{U}[0,\theta]$ distribution, as explained nicely here. This can be a problem even in examining the distribution of bootstrap sample means if the underlying sample is small or does not represent the entire population well.

The "empirical bootstrap" as described in the class notes to which you linked gets around this problem by using the sample mean as an estimate of the population mean, and directly examining the differences of multiple bootstrap sample means from that estimate of the population mean. As described in those notes, such relative variation about the mean is much less sensitive to discrepancies between the sample distribution and the underlying distribution.

This also is a problem for estimating p-values from percentile bootstraps. If the null hypothesis doesn't hold, then bootstrap sampling is from a population described by the alternate hypothesis, not by the null hypothesis. Yet frequentist calculations of p-values are based on what would be expected if the null hypothesis holds. In some circumstances, the percentile bootstrap can still serve for these purposes. As AdamO explains in this answer, there is a critical assumption needed about how the underlying distribution changes between populations described by the null and alternate hypotheses:

But one important assumption is that such a distribution is pivotal. This means that if the underlying parameter changes, the shape of the distribution is only shifted by a constant, and the scale does not necessarily change. This is a strong assumption!

The percentile bootstrap, happily, is only one of several ways to use bootstrapping to estimate CIs or p-values. In addition to the "empirical bootstrap," some are described in the answer from AdamO I linked above and in other answers on that page. This page provides other perspectives and links to further information. The boot.ci() function in the R boot package provides 5 bootstrapping methods for CIs (including the percentile method). For p-values you also can try other resampling techniques. For example, permutation testing can repeatedly shuffle class labels to represent populations described by the null hypothesis. This answer explains several types of resampling.

So "the percentile bootstrap should never be used" might be a bit of overstatement. It certainly works for estimating quantiles (like the median) closer to the center of a distribution. Better to say that it can lead to problems and that it never needs to be used for CIs or p-values.

EdM
  • 101.5k
  • 11
  • 102
  • 303