Why not always use bootstrap CIs?

Question

I was wondering how bootstrap CIs (and BCa in barticular) perform on normally-distributed data. There seems to be lots of work examining their performance on various types of distributions, but could not find anything on normally-distributed data. Since it seems an obvious thing to study first, I suppose the papers are just too old.

I did some Monte Carlo simulations using the R boot package and found bootstrap CIs to be in agreement with exact CIs, although for small samples (N<20) they tend to be a bit liberal (smaller CIs). For large enough samples, they are essentially the same.

This makes me wonder whether there is any good reason not to always use bootstrapping. Given the difficulty of assessing whether a distribution is normal, and the many pitfalls behind this, it seems reasonable not to decide and report bootstrap CIs irrespective of the distribution. I understand the motivation for not using non-parametric tests systematically, since they have less power, but my simulations tell me this is not the case for bootstrap CIs. They are even smaller.

A similar question that bugs me is why not always use the median as the measure of central tendency. People often recommend to use it to characterize non normally-distributed data, but since the median is the same as the mean for normally-distributed data, why make a distinction? It would seem quite beneficial if we could get rid of procedures for deciding whether a distribution is normal or not.

I am very curious about your thoughts on these issues, and whether they have been discussed before. References would be highly appreciated.

Thanks!

Pierre

Answer 1

It is beneficial to look at the motivation for the BCa interval and it mechanisms (i.e. the so called "correction factors"). The BCa intervals are one of the most important aspects of the bootstrap because they are the more general case of the Bootstrap Percentile Intervals (i.e. the confidence interval based solely upon the bootstrap distribution itself).

In particular, look at the relationship between BCa intervals and the Bootstrap Percentile Intervals: when the adjustment for acceleration (the first "correction factor") and skewness (the second "correction factor") are both zero, then the BCa intervals revert back to the typical Bootstrap Percentile Interval.

I do not think that it would be a good idea to ALWAYS use bootstrapping. Bootstrapping is a robust technique that has a variety of mechanisms (ex: confidence intervals and there are different variations of the bootstrap for different types of problems such as the wild bootstrap when there is heteroscedasticity) for adjusting for different problems (ex: non-normality), but it is relies upon one crucial assumption: the data accurately represent the true population.

This assumption, although simple in nature, can be difficult to verify especially in the context of small sample sizes (it could be though that a small sample is an accurate reflection of the true population!). If the original sample on which the bootstrap distribution (and hence all of the results that follow from it) is not adequately accurate, then your results (and hence your decision based upon those results) will be flawed.

CONCLUSION: There is a lot of ambiguity with the bootstrap and you should exercise caution before applying it.

Answer 2

This is a situation like a lot of situations that arise when comparing fully nonparametric methods with parametric methods that rely on broad assumptions (e.g., distribution with finite variance leading to the CLT). Assuming that both methods are constructed appropriately, we usually find three things: (1) the parametric method usually works better than the nonparametric method on small sample problems where the underlying assumptions hold; (2) the nonparametric method usually works better than the parametric method on small sample problems where the underlying assumptions for the parametric method are substantially violated; and (3) when the sample size gets large, both methods work about equally well.

Under this circumstance, some practitioners do indeed prefer to always use nonparametric methods. Such practitioners typically prefer to use methods that have minimal assumptions and they are suspicious of making statistical assumptions to facilitate analysis in cases where the dataset is small. That is a perfectly reasonable position to take, so if you prefer to always use bootstrap CIs, my view is that that is a defensible position. Having said that, you should be careful not to exaggerate the assumptions made by other methods. The standard CIs for means, variances, etc., do not require us to assume that the underlying data is normal --- rather, we assume that the underlying data is such that we can apply the CLT so that important sample quantities (e.g., sample mean) are roughly normally distributed over samples that are not too small.

I understand the motivation for not using non-parametric tests systematically, since they have less power, but my simulations tell me this is not the case for bootstrap CIs. They are even smaller.

Smaller is not necessarily good on its own. Smaller CIs are good if the coverage level is accurate. If the CI is too small, such that the actual coverage level is less than the confidence level, that is bad.

If you would like to do a detailed simulation analysis comparing the bootstrap CI to other CIs, I recommend you look at the width of the CIs but also the proportion of the time the true parameter value falls within the CI in the simulations. Ideally, you want the coverage proportion to be the same as the confidence level, but if the confidence level is an underestimate of the true coverage probability, that is not a catastrophic problem. If you do a large simulation study over appropriate cases, you should be able to determine whether the competing methods produce CIs with accurate confidence levels, and whether the intervals produced are more/less accurate (i.e., narrower/wider) under different methods.

A similar question that bugs me is why not always use the median as the measure of central tendency. People often recommend to use it to characterize non normally-distributed data, but since the median is the same as the mean for normally-distributed data, why make a distinction?

Again, you seem to be proceeding under the view that standard CIs assume normality of the data, when actually what they assume is much weaker than this --- the standard CI for a population mean only assumes that the underlying distribution of the data has finite variance, such that we can apply the CLT to ensure that the sample mean is roughly normally distributed. In samples that are not too small, the sample mean should be roughly normally distributed, but the underlying data usually is not. Consequently, the sample mean will not generally correspond to the sample median in such cases.

Here is is worth noting that the use of the sample mean or median really depends on what you want to make an inference about. If you want a CI for the population mean then the sample mean is a natural consistent estimator; if you want a CI for the population media then the sample median is a natural consistent estimator. In both cases there are applicable CLT results that say that these quantities are roughly normally distributed under weak assumptions for samples that are not too small. Nevertheless, other than for underlying symmetric distributions, these two things do not usually correspond.

Answer 3

The other day, I came across a situational constraint where bootstrap analysis would not work on my presumed normally distributed sample.

I was at the park with my four-year old daughter who started gathering acorns like they were treasures. Her hands were quickly full, so I gestured that it would be okay to deposit the acorns in my pocket. Well she turned my pants into a sumo suit, and we ended up going home with about 300 acorns.

When we got home, I starting wondering what we could learn about the population of fallen acorns from the sample. I weighed the whole sample on a kitchen scale which came out to be about 1150g. Then I started thinking about what this sample might be able to say about the tree's population of acorns. The first thing that came to mind was doing a bootstrap analysis.

However, the equipment I have on hand has an accuracy of 1g. I couldn't just weigh one acorn at a time, since an acorn weighing in at 4g might actually weigh 3g, 4g or 5g. In order to reduce the amount of scale error. I figured the per-acorn scale accuracy would only be off by a tiny amount if I weighed each sample group together. But this group weighing constraint means it was not possible to introduce sampling with replacement, since I could only weigh the acorns as a group. Apart from investing in a better scale it seemed the best option might be weighing 20 random acorns say 25 times. Then using the average of these sample means to approximate a population mean and make some inferences. Sampling without replacement seems like the only option under these conditions.

Answer 4

OP:

This makes me wonder whether there is any good reason not to always use bootstrapping. Given the difficulty of assessing whether a distribution is normal...

Traditional parametric methods rely on the CLT. The data don't have to be Normal, but the sampling distribution should be (asymptotically) Normal.

Alas, bootstrap methods typically also have similar assumptions. The data don't have to be Normal, but the sampling distribution of $\sqrt{n}(\hat\theta-\theta)$ has to be well-defined and well-behaved in order for us to guarantee that the bootstrap works (asymptotically).

See Larry Wasserman's notes:

The bootstrap does not always work. It can fail for a variety of reasons such as when the dimension is high or when [the estimator] is poorly behaved.

So we can't guarantee that the bootstrap will save us from needing a CLT. On the other hand, if a CLT is appropriate, some bootstrap methods can be "second-order accurate." In other words, if your sampling distribution of interest has an asymptotic approximation and you'd rely on the CLT anyway, some kinds of bootstrapping might get you there slightly "faster" (better approximation at lower $n$) and/or with CI coverage closer to nominal. See Section 3 of Davidson, Hinkley, Young (2003), "Recent Developments in Bootstrap Methodology".

---

OP:

A similar question that bugs me is why not always use the median as the measure of central tendency.

The median is one of those estimators that aren't always well-behaved -- whether you're using bootstrap or other methods.

I once was working with a fairly large survey dataset, where one of the questions was about income. We tried to do exactly what you suggest: focus on the median rather than the mean, and use a bootstrap approach to get the CI (though I don't think it was BCa).

It turned out that many respondents had rounded their income to exactly \$50,000. So many, in fact, that in EVERY bootstrap sample the median was also \$50,000! Our bootstrap SE was 0 and our bootstrap CI was (\$50k, \$50k).

We ended up using one of the much older nonparametric CIs for a median, based simply on order statistics of the sample, which (thankfully) gave a reasonable result.