Can bootstrap be seen as a "cure" for the small sample size?

Question

This question has been triggered by something I read in this graduate-level statistics textbook and also (independently) heard during this presentation at a statistical seminar. In both cases, the statement was along the lines of "because the sample size is pretty small, we decided to perform estimation via bootstrap instead of (or along with) this parametric method $X$".

They didn't get into the details, but probably the reasoning was as follows: method $X$ assumes the data follow a certain parametric distribution $D$. In reality the distribution is not exactly $D$, but it's ok as long as the sample size is large enough. Since in this case the sample size is too small, let's switch to the (non-parametric) bootstrap that doesn't make any distributional assumptions. Problem solved!

In my opinion, that's not what bootstrap is for. Here is how I see it: bootstrap can give one an edge when it's more or less obvious that there are enough data, but there is no closed form solution to get standard errors, p-values and similar statistics. A classic example is obtaining a CI for the correlation coefficient given a sample from a bivariate normal distribution: the closed form solution exists, but it is so convoluted that bootstrapping is simpler. However, nothing implies that bootstrap can somehow help one to get away with a small sample size.

Is my perception right?

If you find this question interesting, there is another, more specific bootstrap question from me:

Bootstrap: the issue of overfitting

P.S. I can’t help sharing one egregious example of the “bootstrap approach”. I am not disclosing the author’s name, but he is one of the older generation “quants” who wrote a book on Quantitative Finance in 2004. The example is taken from there.

Consider the following problem: suppose you have 4 assets and 120 monthly return observations for each. The goal is to construct the joint 4-dimensional cdf of yearly returns. Even for a single asset, the task appears hardly attainable with only 10 yearly observations, let alone the estimation of 4-dimensional cdf. But not to worry, the “bootstrap” will help you out: take all of the available 4-dimensional observations, resample 12 with replacement and compound them to construct a single “bootstrapped” 4-dimensional vector of annual returns. Repeat that 1000 times and, lo and behold, you got yourself a “bootstrap sample” of 1000 annual returns. Use this as an i.i.d. sample of size 1000 for the purpose of cdf estimation, or any other inference that can be drawn from a thousand –year history.

Answer 1

I remember reading that using the percentile confidence interval for bootstrapping is equivalent to using a Z interval instead of a T interval and using $n$ instead of $n-1$ for the denominator. Unfortunately I don't remember where I read this and could not find a reference in my quick searches. These differences don't matter much when n is large (and the advantages of the bootstrap outweigh these minor problems when $n$ is large), but with small $n$ this can cause problems. Here is some R code to simulate and compare:

simfun <- function(n=5) {
    x <- rnorm(n)
    m.x <- mean(x)
    s.x <- sd(x)
    z <- m.x/(1/sqrt(n))
    t <- m.x/(s.x/sqrt(n))
    b <- replicate(10000, mean(sample(x, replace=TRUE)))
    c( t=abs(t) > qt(0.975,n-1), z=abs(z) > qnorm(0.975),
        z2 = abs(t) > qnorm(0.975), 
        b= (0 < quantile(b, 0.025)) | (0 > quantile(b, 0.975))
     )
}

out <- replicate(10000, simfun())
rowMeans(out)

My results for one run are:

     t      z     z2 b.2.5% 
0.0486 0.0493 0.1199 0.1631 

So we can see that using the t-test and the z-test (with the true population standard deviation) both give a type I error rate that is essentially $\alpha$ as designed. The improper z test (dividing by sample standard deviation, but using Z critical value instead of T) rejects the null more than twice as often as it should. Now to the bootstrap, it is rejecting the null 3 times as often as it should (looking if 0, the true mean, is in the interval or not), so for this small sample size the simple bootstrap is not sized properly and therefore does not fix problems (and this is when the data is optimally normal). The improved bootstrap intervals (BCa etc.) will probably do better, but this should raise some concern about using bootstrapping as a panacea for small sample sizes.

Answer 2

Other answers criticise the performance of bootstrap confidence intervals, not bootstrap itself. This is a different problem.

If your context satisfy the regularity conditions for the convergence of the bootstrap distribution (convergence in terms of the number of bootstrap samples), then the method will work if you use a large enough bootstrap sample.

In case you really want to find issues of using nonparametric bootstrap, here are two problems:

(1) Issues with resampling.

One of the problems with bootstrap, either for small or large samples, is the resampling step. It is not always possible to resample while keeping the structure (dependence, temporal, ...) of the sample. An example of this is a superposed process.

Suppose that there are a number of independent sources at each of which events occur from time to time. The intervals between successive events at any one source are assumed to be independant random variables all with the same distribution, so that each source constitutes a renewal process of a familiar type. The outputs of the sources are combined into one pooled output.

How would you resample while keeping the dependence unknown structure?

(2) Narrow bootstrap samples and bootstrap confidence intervals for small samples.

In small samples the minimum and maximum of the estimators for each subsample may define a narrow interval, then the right and left end points of any confidence intervals will be very narrow (which is counterintuitive given the small sample!) in some models.

Suppose that $x_1,x_2\sim \text{Exp}(\lambda)$, where $\lambda>0$ is the rate. Using the profile likelihood you can obtain an approximate confidence interval (the 95% approximate confidence interval is the 0.147-level profile likelihood interval) as follows:

set.seed(1)
x <- rexp(2,1)
# Maximum likelihood estimator
1/mean(x)

# Profile likelihood: provides a confidence interval with right-end point beyond the maximum inverse of the mean
Rp <- Vectorize(function(l) exp(sum(dexp(x,rate=l,log=T))-sum(dexp(x,rate=1/mean(x),log=T))))

curve(Rp,0,5)
lines(c(0,5),c(0.147,0.147),col="red")

This method produces a continuous curve from where you can extract the confidence interval. The maximum likelihood estimator of $\lambda$ is $\hat{\lambda}=2/(x_1+x_2)$. By resampling, there are only three possible values that we can obtain for this estimator, whose maximum and minimum define the bounds for the corresponding bootstrap confidence intervals. This may look odd even for large bootstrap samples (you don't gain much by increasing this number):

library(boot)
set.seed(1)
x <- rexp(2,1)
1/mean(x)
# Bootstrap interval: limited to the maximum inverse of the mean
f.boot <- function(data,ind) 1/mean(data[ind])
b.b <- boot(data=x, statistic=f.boot, R=100000)
boot.ci(b.b, conf = 0.95, type = "all")
hist(b.b$t)

In this case, the closer $x_1$ and $x_2$ are, the narrower the bootstrap distribution is, and consequently the narrower the confidence interval (which might be located far from the real value). This example is, in fact, related to the example presented by @GregSnow, although his argument was more empirical. The bounds I mention explain the bad performance of all the bootstrap confidence intervals analysed by @Wolfgang.

Answer 3

If you are provided with small sample size (as a sidelight, what is "small" seems to depend on some underlying customary rule in each research field), no bootstrap will do the magic. Assuming a database contains three observations for each of the two variables under investigation, no inference will make sense. In my experience, non-parametric bootstrap (1,000 or 10,000 replications) works well in replacing t-test when sample distributions (at least 10-15 observations each) are skewed and therefore the prerequisites for the usual t-test are not satisfied. Besides, regardless the number of observations, non-parametric bootstrap may be a mandatory choice when data are positively skewed, as it always happens for health care costs. Other interesting applications for non-parametric bootstrap relate to standard errors calculation for coefficients included in regressions and panel datasets.

Answer 4

Bootstrap works well in small sample sizes by ensuring the correctness of tests (e.g. that the nominal 0.05 significance level is close to the actual size of the test), however the bootstrap does not magically grant you extra power. If you have a small sample, you have little power, end of story.

Parametric (linear models) and semiparametric (GEE) regressions tend to have poor small sample properties... the former as a consequence of large dependence on parametric assumptions, the latter because of magnification of robust standard error estimates in small samples. Bootstrapping (and other resampling based tests) performs really well in those circumstances.

For prediction, bootstrapping will give you better (more honest) estimates of internal validity than split sample validation.

Bootstrapping often times gives you less power as a consequence of inadvertently correcting mean imputation procedures / hotdecking (such as in fuzzy matching). Bootstrapping has been erroneously purported to give more power in matched analyses where individuals were resampled to meet the sufficient cluster size, giving bootstrapped matched datasets with a greater $n$ than the analysis dataset.