# Bootstrap confidence intervals - how many replications to choose?

I applied a bootstrap-process to calculate confidence intervalls for the paramters of a multiple lineare regression.

In R it's pretty simple to implement (functions: 'boot' and 'boot.ci') but I still have two comprehension problems:

1. Why does it make sense to perform a bootstrap procedure before calculating the confidence intervals? Will they be more precise? And if so, can anyone explain why?
2. How can I decide which number of replications is a good number for calculating confidence intervalls? 100? 1000? 10000?

I would really appreciate any help!

• If you use a normal approximation on your bootstrap estimates you can get away doing much less bootstrap replicates though (e.g. 100) than when you use the percentile method (in which case >10 000 is recommended). Jun 5, 2019 at 21:39

Why does it make sense to perform a bootstrap procedure before calculating the confidence intervals? Will they be more precise? And if so, can anyone explain why?

You can calculate bootstrap confidence intervals for complex situations, i.e. properties ("statistics") that are not easily accessible analytically. I'm thinking of things like bootstrapping generalization error of a predictive model*.

In other words, bootstrapping may still be possible in situations where you have no good assumption which distribution to base your confidence intervals on.

The choice parametric (analytical confidence interval based on known distribution) vs. non-parametric bootstrap is a trade-off:

• good parametric statistics will be more precise. But they may be totally off if the assumptions are violated (i.e. the distribution you chose was not appropriate).
• bootstrap is less precise (for a given number of original cases) but does not rely on particular distribution assumptions, so there's less danger of getting that part wrong*.

How can I decide which number of replications is a good number for calculating confidence intervalls? 100? 1000? 10000?

@MartenBuuis already gave you some idea how to approach this question. Here's another, very pragmatic one:

1. Bootstrap, say, with nboot = 100 replications.
2. repeat this 10 times
3. check variability of the bootstrap results.
4. if the variation you observe over the repetitions of the bootstrapping calculation is acceptable for your application, fuse the 10x100 calculations and use the result of that nboot = 10x100 = 1000 replications.
If they are not sufficiently precise, fuse the 10x100 calculations, go back to step 1 and 2 with nboot = 1000 replications.

You get the idea.

• If you use a normal approximation on your bootstrap estimates you can get away doing much less bootstrap replicates though (e.g. 100) than when you use the percentile method (in which case >10 000 is recommended). Jun 5, 2019 at 21:39

Let's take the simplest case of using just the percentiles to compute the confidence interval. In that case you repeatedly sample with replacement from your data, compute your statistic in each of these samples and store those estimates. The 2.5th percentile of those stored estimates represents the lower bound and the 97.5th percentile the upper bound of a 95% confidence interval.

If you have 200 replications that lower bound will be based on the 5th smallest estimate and the highest on the 5th highest. That will be way too small for my taste. My default is 20,000 replications, so the lower bound would be based on the 500th smallest estimate and the upper bound on the 500th largest estimate. The default is nothing but a starting point, and I will often choose another number depending on the exact circumstances.

• Thanks for your answer! But what is the advantage of the resampling? Why can't I just take the 2.5th and 97.5th percentile from my original data? So, when it comes to deciding what number of replications is appropriate, you recommend selecting the largest possible value (if problems with performance are ignored)? Jul 17, 2018 at 11:59
• @MarcelGrimmer: you can bootstrap the confidence interval for a statistic of your data (say, mean, median, variance, or some number that is far more complicated to compute, e.g. generalized prediction error of a model fit to your data). You can report percentiles only for your data itself. Also note that saying, the 2.5th percentile of x is in this confidence interval is not the same as saying the 2.5th percentile of our observations of x is... Jul 17, 2018 at 12:13
• @cbeleites Thats right - I was wrong. Thanks! Jul 18, 2018 at 13:17
• If you use a normal approximation on your bootstrap estimates you can get away doing much less bootstrap replicates though (e.g. 100) than when you use the percentile method (in which case >10 000 is recommended). Jun 5, 2019 at 21:39
• @TomWenseleers true, but you often use the bootstrap because the normal approximation does not apply.... Jun 6, 2019 at 7:22

According to Efron (the "inventor" of the boostrap technique), you should make 1600 replicas. I have no other clue about where this number comes from, except that its square root is 40, an easy number to divide by. I suggest you go like in any other Monte-Carlo. Try 1600, then increase the bootstrap samples until it stabilizes.

The bootstrap was introduced to compute confidence intervals in case the distribution of the v.a.r is unknown or not technically computable, because of outliers or skew. The bootstrap replaces the theoretical computations of the confidence interval by a measure of simulated samples. So the confidence intervals should be the same.

Note however that all statistical indicators are not equal in front of he boostrap. A average (or a mean) will require less samples than a maximum, or a 1% percentile for example.

## 1. Bootstrap before calculating CIs

Not sure if I understood your question correctly, but if you were asking, ‘Should I be using bootstrap to compute the CIs’, then, the missing part of the question is, ‘bootstrap instead of what’, ‘more precise’ – ‘precise compared to what and in which sense’.

There are multiple ways to construct the CIs:

1. Make an assumption about the sampling distribution of the estimator the for every sample size $$n$$. This is a very strong assumption; however, people have done it many times in the past: they often assume the $$t$$ distribution (for sample sizes at least 2). This is the classical frequentist (Fisher) inference.
2. Assume that nothing is known anything about the sampling distribution for any given $$n$$, but as $$n \to\infty$$, you know the distribution: $$n^q (\hat \theta_n - \theta_0) \xrightarrow[n\to\infty]{d} \mathcal{N}(0, V)$$, where $$q$$ is the rate of convergence ($$q=1/2$$ for most parametric estimators, $$q = 1/5$$ or lower for non-parametric ones, $$q=2/5$$ for the smoothed Manski estimator etc.). Then, you just look up the table of Gaussian critical values. The problem is, estimating $$V$$ is sometimes non-trivial.
3. Estimate the critical value by bootstrapping. This is where the consistency of bootstrap is required (i.e. no parameter on the boundary of the parameter space, the same rate of convergence of the original and bootstrap estimators etc. – in general, the failure of $$\sup_{u\in \mathbb{R}} |\mathrm{CDF}_{\sqrt{n}(\hat\theta^*_n - \hat\theta_n)} (u) - \mathrm{CDF}_{\sqrt{n}(\hat\theta_n - \theta_0)} (u)| \xrightarrow[n\to\infty]{\mathbb{P}} 0$$ (in Efron’s notation), which can happen due to a multitude of reasons).

So if bootstrap works in the sense of the (rather technical) condition described above, then, depending on some extra conditions (such as requiring finite estimator variances), it can beat the asymptotic confidence intervals (as well as the variance estimators, $$p$$-values – basically, any functional of the estimator distribution) in the sense of the approximation error. Assume that $$\mathbb{E} (\hat\theta_n - \theta_0)^2 < \infty$$ (which rules out certain estimators, like the IV estimator that is the ratio of two Gaussians) and that the sampling distribution of $$\hat\theta_n$$ is symmetrical. Then, bootstrap is ‘better’ in the following sense:

$$\sup_{u\in \mathbb{R}} |\mathrm{CDF}_{\sqrt{n}(\hat\theta_n - \hat\theta_0) / \mathrm{SE} \hat\theta_n} (u) - \Phi(u)| = O(1/\sqrt{n}),$$

$$\sup_{u\in \mathbb{R}} |\mathrm{CDF}_{\sqrt{n}(\hat\theta_n^* - \hat\theta_n) / \mathrm{SE}^* \hat\theta^*_n} (u) - \mathrm{CDF}_{\sqrt{n}(\hat\theta_n - \theta_0) / \mathrm{SE} \hat\theta_n} (u)| = O_p(1/n),$$

where $$\Phi$$ is the CDF of the standard normal distribution.

Or course it does not guarantee that the capital O in a specific given application is not going to bring the refinement, and of course, depending on the smoothness of the bootstrapped quantity (bias, or variance, or CI, or p-value) and the bootstrap type, the refinement may or may not exist – however, if you are worrying that the bootstrap is going to be less reliable than asymptotic confidence intervals – probably not. Bootstrap does a much better job on reproducing the shape of the sampling distribution (on which one should really be doing inference), which is why if your method belongs to a broad class of estimators for which bootstrap works and you can afford running the bootstrap sufficiently many times, then, bootstrap should be more trustworthy in the sense of capturing the extra moments of the sampling distribution uncertainty (as opposed to the first 2 moments of the asymptotic Gaussian approximation). Depending on how poorly the Weak Law of Large Numbers is working when the numbers are not large enough, bootstrap may simply unveil it to the researcher.

NB. Bootstrap is an asymptotic method in the sense that it still relies in most aspects on the number of observations $$n \to \infty$$. Bootstrap does not improve theoretical properties of statistical tests, and if $$n=8$$, the question would be, ‘is scientific method really applicable?’ or ‘aren’t we making conclusions about random data features, not the true underlying relationships?’. If the theoretical power of one’s test is low or there are extra complications in the form of departures from the ‘randomised controlled trial’ setting (as well as ‘independent identically distributed’), bootstrap won’t help, and the paper will be rejected.

## 2.1 Number of replications (practical advice)

A large number of replications B is required to say that the finite-sample Monte-Carlo approximation replicates the asymptotic (in B) bootstrap distribution of the object of interest closely enough. This means that there is no penalty (other than increased computation time) to doing more replications in one experimental setting. Unless one is studying the theoretical properties of bootstrap (e.g. nested bootstrap, second-order bootstrap, bootstrapping new estimators etc.), then, the infallible rule is ‘more is better’.

In case one does not have deep bootstrap knowledge, here is a quick recommendation (that should hopefully stay relevant for another decade).

1. B >= 1000, otherwise your paper will be rejected with something like ‘We are not in the Pentium-II era’ from Referee 2.
2. Ideally, B >= 10000; try to do it if your computer can handle it.

Here is where most researchers may stop. However, if the researcher suspects that their sampling distribution may be irregular and discrepancies between the true and simulated distribution are large, then, we may check some features of the bootstrap distribution to determine how close we are to it (as a function of $$B$$). The seminal paper is Andrews & Buchinsky (2000, Econometrica). Here are the extra steps to make any picky referee shut up:

1. You could check if your B yields the desired probability $$1-\tau$$ of achieving the desired relative accuracy $$r$$ of the bootstrapped quantity of interest for some common level (e.g. $$r= 5\%$$ and $$\tau=5\%$$).
2. If not, increase B to the value dictated by the A&B 3-stage procedure described below.
3. In general, for any actual accuracy of your bootstrapped quantity, to increase the desired relative accuracy by a factor of k, increase B by a factor of $$k^2$$.

## 2.2 A data-driven theory-backed procedure

There is a data-driven method of choosing B: do some small number of bootstrap replications, see how stable or noisy the estimator is, and then, based on some target accuracy measure, increase the number of replications until you are sure that this resampling-related error has reached a certain lower bound with a chosen certainty. Our helper here is the Weak Law of Large Numbers where the asymptotics are in B. To be more specific, B is chosen depending on the user-chosen bound on the relative deviation measure of the Monte-Carlo approximation of the quantity of interest based on B simulations. This quantity can be standard error, p-value, confidence interval, or bias correction. The closeness is the relative deviation $$R^*$$ of the B-replication bootstrap quantity from the infinite-replication quantity (or, to be more precise, the one that requires $$n^n$$ replications): $$R^* := (\hat\lambda_B - \hat\lambda_\infty)/\hat\lambda_\infty$$. The idea is, find such B that the actual relative deviation of the statistic of interest be less than a chosen bound (usually 5%, 10%, 15%) with a specified high probability $$1-\tau$$ (usually $$\tau = 5\%$$ or $$10\%$$). Then,

$$\sqrt{B} \cdot R^* \xrightarrow{d} \mathcal{N}(0, \omega),$$

where $$\omega$$ can be estimated using a relatively small (usually 200–300) preliminary bootstrap sample that one should be doing in any case.

Here is the general formula for the number of necessary bootstrap replications $$B$$:

$$B \ge \omega \cdot (Q_{\mathcal{N}(0, 1)}(1-\tau/2) / r)^2,$$

where r is the maximum allowed relative discrepancy (i.e. accuracy), $$1-\tau$$ is the probability that this desired relative accuracy bound has been achieved, $$Q_{\mathcal{N}(0, 1)}$$ is the quantile function of the standard Gaussian distribution, and $$\omega$$ is the asymptotic variance of $$R$$*. The only unknown quantity here is $$\omega$$ that represents the variance due to simulation randomness.

The general 3-step procedure for choosing B is like this:

1. Compute the approximate preliminary number $$B_1 := \lceil \omega_1 (Q_{\mathcal{N}(0, 1)}(1-\tau/2) / r)^2 \rceil$$, where $$\omega_1$$ is a very simple theoretical formula from Table III in Andrews & Buchinsky (2000, Econometrica).
2. Using these $$B_1$$ samples, compute an improved estimate $$\hat\omega_{B_1}$$ using a formula from Table IV (ibid.).
3. With this $$\hat\omega_{B_1}$$ compute $$B_2 := \lceil\hat\omega_{B_1} (Q_{\mathcal{N}(0, 1)}(1-\tau/2) / r)^2 \rceil$$ and take $$B_{\mathrm{opt}} := \max(B_1, B_2)$$.

If necessary, this procedure can be iterated to improve the estimate of $$\omega$$, but this 3-step procedure as it is tends to yield already conservative estimates that ensure that the desired accuracy has been achieved. This approach can be vulgarised by taking some fixed $$B_1 = 1000$$, doing 1000 bootstrap replications in any case, and then, doing steps 2 and 3 to compute $$\hat\omega_{B_1}$$ and $$B_2$$.

Example (Table V, ibid.): to compute a bootstrap 95% CI for the linear regression coefficients, in most practical settings, to be 90% sure that the relative CI length discrepancy does not exceed 10%, 700 replications are sufficient in half of the cases, and to be 95% sure, 850 replications. However, requiring a smaller relative error (5%) increases B to 2000 for $$\tau=10\%$$ and to 2700 for $$\tau=5\%$$.

This agrees with the formula for B above. If one seeks to reduce the relative discrepancy r, by a factor of k, the optimal B goes up roughly by a factor of $$k^2$$, whilst increasing the confidence level that the desired closeness is reached merely changes the critical value of the standard normal (1.96 → 2.57 for 95% → 99% confidence).