# Can bootstrap be seen as a "cure" for the small sample size?

This question has been triggered by something I read in this graduate-level statistics textbook and also (independently) heard during this presentation at a statistical seminar. In both cases, the statement was along the lines of "because the sample size is pretty small, we decided to perform estimation via bootstrap instead of (or along with) this parametric method $X$".

They didn't get into the details, but probably the reasoning was as follows: method $X$ assumes the data follow a certain parametric distribution $D$. In reality the distribution is not exactly $D$, but it's ok as long as the sample size is large enough. Since in this case the sample size is too small, let's switch to the (non-parametric) bootstrap that doesn't make any distributional assumptions. Problem solved!

In my opinion, that's not what bootstrap is for. Here is how I see it: bootstrap can give one an edge when it's more or less obvious that there are enough data, but there is no closed form solution to get standard errors, p-values and similar statistics. A classic example is obtaining a CI for the correlation coefficient given a sample from a bivariate normal distribution: the closed form solution exists, but it is so convoluted that bootstrapping is simpler. However, nothing implies that bootstrap can somehow help one to get away with a small sample size.

Is my perception right?

If you find this question interesting, there is another, more specific bootstrap question from me:

Bootstrap: the issue of overfitting

P.S. I can’t help sharing one egregious example of the “bootstrap approach”. I am not disclosing the author’s name, but he is one of the older generation “quants” who wrote a book on Quantitative Finance in 2004. The example is taken from there.

Consider the following problem: suppose you have 4 assets and 120 monthly return observations for each. The goal is to construct the joint 4-dimensional cdf of yearly returns. Even for a single asset, the task appears hardly attainable with only 10 yearly observations, let alone the estimation of 4-dimensional cdf. But not to worry, the “bootstrap” will help you out: take all of the available 4-dimensional observations, resample 12 with replacement and compound them to construct a single “bootstrapped” 4-dimensional vector of annual returns. Repeat that 1000 times and, lo and behold, you got yourself a “bootstrap sample” of 1000 annual returns. Use this as an i.i.d. sample of size 1000 for the purpose of cdf estimation, or any other inference that can be drawn from a thousand –year history.

• I hear or read similar sentiments to your first paragraph, but I tend to disagree. I don't usually see bootstrapping as necessarily useful in small samples. Its justification is asymptotic/large sample, and in many cases in practice its small sample performance is problematic; coverage of intervals, for example, is often quite different from the nominal rate. It's another useful tool but like many others that have been heralded over the years, it's not the panacea some people imagine it to be. I really don't think it's a solution to very small samples. Aug 17 '14 at 0:24
• In Simon Sheather's book (Figure 3.40) there is a flow chart for regression that suggests the bootstrap for inference when the sample size is not large and errors are not normally distributed. Aug 20 '14 at 0:24
• The answer to the question in the fourth paragraph is yes (or at least you would get a credit for your argument if you gave this answer on a test in my class). Have you checked this discussion? In my answer, I gave an explanation of the approximations that the bootstrap makes, and gave a reference to the blow-your-mind paper that every bootstrapper should read to be aware of the limitations of the method. It would be nice to see references to the books/papers/presentations that triggered your question Aug 27 '14 at 17:00
• Doesn't the bootstrap just give you an extra term in an edgeworth expansion - like a normal approximation with a correction term? Jun 2 '16 at 13:29

I remember reading that using the percentile confidence interval for bootstrapping is equivalent to using a Z interval instead of a T interval and using $n$ instead of $n-1$ for the denominator. Unfortunately I don't remember where I read this and could not find a reference in my quick searches. These differences don't matter much when n is large (and the advantages of the bootstrap outweigh these minor problems when $n$ is large), but with small $n$ this can cause problems. Here is some R code to simulate and compare:

simfun <- function(n=5) {
x <- rnorm(n)
m.x <- mean(x)
s.x <- sd(x)
z <- m.x/(1/sqrt(n))
t <- m.x/(s.x/sqrt(n))
b <- replicate(10000, mean(sample(x, replace=TRUE)))
c( t=abs(t) > qt(0.975,n-1), z=abs(z) > qnorm(0.975),
z2 = abs(t) > qnorm(0.975),
b= (0 < quantile(b, 0.025)) | (0 > quantile(b, 0.975))
)
}

out <- replicate(10000, simfun())
rowMeans(out)


My results for one run are:

     t      z     z2 b.2.5%
0.0486 0.0493 0.1199 0.1631


So we can see that using the t-test and the z-test (with the true population standard deviation) both give a type I error rate that is essentially $\alpha$ as designed. The improper z test (dividing by sample standard deviation, but using Z critical value instead of T) rejects the null more than twice as often as it should. Now to the bootstrap, it is rejecting the null 3 times as often as it should (looking if 0, the true mean, is in the interval or not), so for this small sample size the simple bootstrap is not sized properly and therefore does not fix problems (and this is when the data is optimally normal). The improved bootstrap intervals (BCa etc.) will probably do better, but this should raise some concern about using bootstrapping as a panacea for small sample sizes.

• Your last sentence made me curious, so I tried this out with your example. I used the boot package and had it provide all the possible intervals: the first order normal approximation interval, the basic bootstrap interval, the studentized bootstrap interval, the bootstrap percentile interval (which your code already provides), and the adjusted bootstrap percentile (BCa) interval. Results were essentially all the same (rejection rates around .16-.17), except for the studentized interval, which had a nominal (.05) rejection rate (which makes sense, since it is more like the standard t-test). Aug 20 '14 at 22:38
• @Wolfgang, thanks for doing the additional simulations. I am a little surprised that the other intervals did not do a little better, but not surprised that they did not meet the 0.05 level. Aug 21 '14 at 17:35
• Yes, I was a bit surprised, too! Sometimes our intuition fails. That's why it is so wonderful to be able to write up a little script in R within a few minutes, have it run 10,000 \times 10,000 iterations (that took another couple minutes to complete), and be able to check things. And your example nicely shows that the bootstrap can perform quite badly with small samples even when things are "nice" (i.e., the data are in fact normal). Aug 23 '14 at 9:34
• Actually, the problem seen in this simulation is exactly the one I just asked about here and answered in the comments by @whuber: stats.stackexchange.com/questions/323455/…. Hence, I don't think the problem is the sample size, but the violation of independence for parametric inference within each sample. Jan 18 '18 at 22:59
• Your comment on the percentile confidence interval being equivalent to using $z$ instead of $t$ is from here, just for the record. May 11 '18 at 4:12

If you are provided with small sample size (as a sidelight, what is "small" seems to depend on some underlying customary rule in each research field), no bootstrap will do the magic. Assuming a database contains three observations for each of the two variables under investigation, no inference will make sense. In my experience, non-parametric bootstrap (1,000 or 10,000 replications) works well in replacing t-test when sample distributions (at least 10-15 observations each) are skewed and therefore the prerequisites for the usual t-test are not satisfied. Besides, regardless the number of observations, non-parametric bootstrap may be a mandatory choice when data are positively skewed, as it always happens for health care costs. Other interesting applications for non-parametric bootstrap relate to standard errors calculation for coefficients included in regressions and panel datasets.

• bootstrap may be a mandatory choice when data are positively skewed - why not use lognormal or similar skewed distribution instead? SAS and R have lots of choices. Aug 20 '14 at 14:03
• James' reply worths consideration. However, the usual problem I face with logging cost data is the back-transformation to the original metric before presenting results to decision-makers. You can avoid this problem with non-parametric bootstrap. Aug 21 '14 at 9:24
• Carlo, since you have experience with log transformation, you may wish to answer another question of mine (there is a 50 point bounty) : stats.stackexchange.com/a/111932/54099 Aug 21 '14 at 14:29
• James, I have given a temptative answer to your question some minutes ago. Aug 25 '14 at 16:01
• $t$-tests can be corrected for skewed populations, see jstor.org/stable/2286597 -- the proposed corrections worked well with quite skewed populations like $\chi^2_1$ with 10 observations. Aug 27 '14 at 16:49

Other answers criticise the performance of bootstrap confidence intervals, not bootstrap itself. This is a different problem.

If your context satisfy the regularity conditions for the convergence of the bootstrap distribution (convergence in terms of the number of bootstrap samples), then the method will work if you use a large enough bootstrap sample.

In case you really want to find issues of using nonparametric bootstrap, here are two problems:

## (1) Issues with resampling.

One of the problems with bootstrap, either for small or large samples, is the resampling step. It is not always possible to resample while keeping the structure (dependence, temporal, ...) of the sample. An example of this is a superposed process.

Suppose that there are a number of independent sources at each of which events occur from time to time. The intervals between successive events at any one source are assumed to be independant random variables all with the same distribution, so that each source constitutes a renewal process of a familiar type. The outputs of the sources are combined into one pooled output.

How would you resample while keeping the dependence unknown structure?

## (2) Narrow bootstrap samples and bootstrap confidence intervals for small samples.

In small samples the minimum and maximum of the estimators for each subsample may define a narrow interval, then the right and left end points of any confidence intervals will be very narrow (which is counterintuitive given the small sample!) in some models.

Suppose that $x_1,x_2\sim \text{Exp}(\lambda)$, where $\lambda>0$ is the rate. Using the profile likelihood you can obtain an approximate confidence interval (the 95% approximate confidence interval is the 0.147-level profile likelihood interval) as follows:

set.seed(1)
x <- rexp(2,1)
# Maximum likelihood estimator
1/mean(x)

# Profile likelihood: provides a confidence interval with right-end point beyond the maximum inverse of the mean
Rp <- Vectorize(function(l) exp(sum(dexp(x,rate=l,log=T))-sum(dexp(x,rate=1/mean(x),log=T))))

curve(Rp,0,5)
lines(c(0,5),c(0.147,0.147),col="red")


This method produces a continuous curve from where you can extract the confidence interval. The maximum likelihood estimator of $\lambda$ is $\hat{\lambda}=2/(x_1+x_2)$. By resampling, there are only three possible values that we can obtain for this estimator, whose maximum and minimum define the bounds for the corresponding bootstrap confidence intervals. This may look odd even for large bootstrap samples (you don't gain much by increasing this number):

library(boot)
set.seed(1)
x <- rexp(2,1)
1/mean(x)
# Bootstrap interval: limited to the maximum inverse of the mean
f.boot <- function(data,ind) 1/mean(data[ind])
b.b <- boot(data=x, statistic=f.boot, R=100000)
boot.ci(b.b, conf = 0.95, type = "all")
hist(b.b$t)  In this case, the closer$x_1$and$x_2$are, the narrower the bootstrap distribution is, and consequently the narrower the confidence interval (which might be located far from the real value). This example is, in fact, related to the example presented by @GregSnow, although his argument was more empirical. The bounds I mention explain the bad performance of all the bootstrap confidence intervals analysed by @Wolfgang. Bootstrap works well in small sample sizes by ensuring the correctness of tests (e.g. that the nominal 0.05 significance level is close to the actual size of the test), however the bootstrap does not magically grant you extra power. If you have a small sample, you have little power, end of story. Parametric (linear models) and semiparametric (GEE) regressions tend to have poor small sample properties... the former as a consequence of large dependence on parametric assumptions, the latter because of magnification of robust standard error estimates in small samples. Bootstrapping (and other resampling based tests) performs really well in those circumstances. For prediction, bootstrapping will give you better (more honest) estimates of internal validity than split sample validation. Bootstrapping often times gives you less power as a consequence of inadvertently correcting mean imputation procedures / hotdecking (such as in fuzzy matching). Bootstrapping has been erroneously purported to give more power in matched analyses where individuals were resampled to meet the sufficient cluster size, giving bootstrapped matched datasets with a greater$n\$ than the analysis dataset.

• (e.g. that the nominal 0.05 significance level is close to the actual size of the test), - but in the R code above we just saw that the bootstrap doesn't preserve the test size, right? Aug 21 '14 at 14:26
• I see you're referring to the answer provided by Greg Snow. I was surprised by that result. I will need to recheck my facts and edit my answer. Thanks. Aug 21 '14 at 17:27