Bootstrap method- downsides

Question

Can you tell me when doesnt the bootstrap method work? I know that could be outliers, but is there any particular distribution when it doesn't work?

Answer 1

There's two different ways to interpret this question:

1.) Is there any types of data such that bootstrapping standard estimators leads to invalid inference?

2.) Are there any non standard estimators such that bootstrap leads to invalid inference?

The answer to both questions is yes.

In the first case, as you mention in your comments, the Cauchy distribution will cause problems in regards to comparing simple means, as will any $t$ distribution with degrees of freedom $\leq 2$. This is because the variance (in the true population) in these cases is infinite. The validity of the bootstrap depends on the sampled data being approximately distributed approximately the same as the true population. But of course there will not be the case when the variance is infinite, as the variance will be finite in any sample.

The practical implications of these are difficult to imagine, however. In practice, we don't usually think of sampling data from a population with an infinite variance.

A better rule of thumb is to consider that you need to have your sample as a good representation of your population of interest. Hence the issue with outliers: if you have very few, highly influential outliers in your sample, you need to recognize that the distribution of your estimator is very heavily influenced by the tails of the distribution, for which you have very little data. Thus, what the bootstrap tells you about the distribution of your estimator is likely to be inaccurate, as it is highly dependent on an aspect of the population's distribution that you empirically know very little about.

In terms of case 2, perhaps the most well known example of the bootstrap failing is the case of the MLE for the uniform distribution.

Suppose you know that $X_i \sim$ uniform(0, $\theta$). Then the MLE based on samples $x_1, ..., x_n$ is $\max(x_i)$. But clearly, if you try to do a non-parametric bootstrap CI based on resampled $x_i$'s, the maximum value resampled will necessarily be less than or equal to $\max(x_i)$. With probability 1, $\max(x_i) < \theta$. Therefore, your non-parametric CI will not contain $\theta$.

A less trivial example is the Grenander estimator (or strictly monotonic density estimator). See http://arxiv.org/abs/1010.3825. Similarly, the non-parametric maximum likelihood estimator for interval censored data (or NPMLE, a generalization of the better known Kaplan Meier curve) suffers a similar problem. See http://arxiv.org/pdf/1312.6341.pdf.