I was wondering how bootstrap CIs (and BCa in barticular) perform on normally-distributed data. There seems to be lots of work examining their performance on various types of distributions, but could not find anything on normally-distributed data. Since it seems an obvious thing to study first, I suppose the papers are just too old.

I did some Monte Carlo simulations using the R boot package and found bootstrap CIs to be in agreement with exact CIs, although for small samples (N<20) they tend to be a bit liberal (smaller CIs). For large enough samples, they are essentially the same.

This makes me wonder whether there is any good reason not to always use bootstrapping. Given the difficulty of assessing whether a distribution is normal, and the many pitfalls behind this, it seems reasonable not to decide and report bootstrap CIs irrespective of the distribution. I understand the motivation for not using non-parametric tests systematically, since they have less power, but my simulations tell me this is not the case for bootstrap CIs. They are even smaller.

A similar question that bugs me is why not always use the median as the measure of central tendency. People often recommend to use it to characterize non normally-distributed data, but since the median is the same as the mean for normally-distributed data, why make a distinction? It would seem quite beneficial if we could get rid of procedures for deciding whether a distribution is normal or not.

I am very curious about your thoughts on these issues, and whether they have been discussed before. References would be highly appreciated.



  • $\begingroup$ See my question here regarding mean and median: stats.stackexchange.com/questions/96371/… $\endgroup$ – Alexis May 25 '14 at 13:35
  • $\begingroup$ For many problems, resampling is computationally infeasible. For example, if you simply want to calculate CIs for a large 3D matrix, or for a long time series. $\endgroup$ – jona Jun 8 '14 at 19:35

It is beneficial to look at the motivation for the BCa interval and it mechanisms (i.e. the so called "correction factors"). The BCa intervals are one of the most important aspects of the bootstrap because they are the more general case of the Bootstrap Percentile Intervals (i.e. the confidence interval based solely upon the bootstrap distribution itself).

In particular, look at the relationship between BCa intervals and the Bootstrap Percentile Intervals: when the adjustment for acceleration (the first "correction factor") and skewness (the second "correction factor") are both zero, then the BCa intervals revert back to the typical Bootstrap Percentile Interval.

I do not think that it would be a good idea to ALWAYS use bootstrapping. Bootstrapping is a robust technique that has a variety of mechanisms (ex: confidence intervals and there are different variations of the bootstrap for different types of problems such as the wild bootstrap when there is heteroscedasticity) for adjusting for different problems (ex: non-normality), but it is relies upon one crucial assumption: the data accurately represent the true population.

This assumption, although simple in nature, can be difficult to verify especially in the context of small sample sizes (it could be though that a small sample is an accurate reflection of the true population!). If the original sample on which the bootstrap distribution (and hence all of the results that follow from it) is not adequately accurate, then your results (and hence your decision based upon those results) will be flawed.

CONCLUSION: There is a lot of ambiguity with the bootstrap and you should exercise caution before applying it.

  • 2
    $\begingroup$ "If the original sample on which the bootstrap distribution (and hence all of the results that follow from it) is not adequately accurate, then your results (and hence your decision based upon those results) will be flawed." -> but does the bootstrap CI perform worse than the analytic alternative in these cases? $\endgroup$ – jona Jun 8 '14 at 19:36
  • 3
    $\begingroup$ The assumption that the data adequately represents the population is not exclusive to only bootstrapping: it pertains to statistics in general, so if the data is inadequate then any inferences, bootstrap inferences or otherwise, that have been made are misleading (a false assumption implies any conclusion!). $\endgroup$ – mmmmmmmmmm Jun 8 '14 at 21:04
  • 4
    $\begingroup$ So this caveat is not directed at the bootstrap, but about inference, and is not an argument against using the bootstrap over another method, but about putting absolute faith in fallible methods. I don't see how it's relevant in this context then. $\endgroup$ – jona Jun 8 '14 at 21:13
  • $\begingroup$ Sorry, a little late on this thread....@jona: wouldn't this be relevant because bootstrapping will further amplify the sample? So if your sample is an unfortunate misrepresentation of the population then using bootstrapping would send you even further away from the population center? Isn't that a potential argument against relying on bootstrapping systematically? $\endgroup$ – sisdog Jul 18 '17 at 0:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.