# Bootstrap confidence intervals - how many replications to choose?

I applied a bootstrap-process to calculate confidence intervalls for the paramters of a multiple lineare regression.

In R it's pretty simple to implement (functions: 'boot' and 'boot.ci') but I still have two comprehension problems:

1. Why does it make sense to perform a bootstrap procedure before calculating the confidence intervals? Will they be more precise? And if so, can anyone explain why?
2. How can I decide which number of replications is a good number for calculating confidence intervalls? 100? 1000? 10000?

I would really appreciate any help!

• If you use a normal approximation on your bootstrap estimates you can get away doing much less bootstrap replicates though (e.g. 100) than when you use the percentile method (in which case >10 000 is recommended). – Tom Wenseleers Jun 5 '19 at 21:39

Why does it make sense to perform a bootstrap procedure before calculating the confidence intervals? Will they be more precise? And if so, can anyone explain why?

You can calculate bootstrap confidence intervals for complex situations, i.e. properties ("statistics") that are not easily accessible analytically. I'm thinking of things like bootstrapping generalization error of a predictive model*.

In other words, bootstrapping may still be possible in situations where you have no good assumption which distribution to base your confidence intervals on.

The choice parametric (analytical confidence interval based on known distribution) vs. non-parametric bootstrap is a trade-off:

• good parametric statistics will be more precise. But they may be totally off if the assumptions are violated (i.e. the distribution you chose was not appropriate).
• bootstrap is less precise (for a given number of original cases) but does not rely on particular distribution assumptions, so there's less danger of getting that part wrong*.

How can I decide which number of replications is a good number for calculating confidence intervalls? 100? 1000? 10000?

@MartenBuuis already gave you some idea how to approach this question. Here's another, very pragmatic one:

1. Bootstrap, say, with nboot = 100 replications.
2. repeat this 10 times
3. check variability of the bootstrap results.
4. if the variation you observe over the repetitions of the bootstrapping calculation is acceptable for your application, fuse the 10x100 calculations and use the result of that nboot = 10x100 = 1000 replications.
If they are not sufficiently precise, fuse the 10x100 calculations, go back to step 1 and 2 with nboot = 1000 replications.

You get the idea.

• If you use a normal approximation on your bootstrap estimates you can get away doing much less bootstrap replicates though (e.g. 100) than when you use the percentile method (in which case >10 000 is recommended). – Tom Wenseleers Jun 5 '19 at 21:39

Let's take the simplest case of using just the percentiles to compute the confidence interval. In that case you repeatedly sample with replacement from your data, compute your statistic in each of these samples and store those estimates. The 2.5th percentile of those stored estimates represents the lower bound and the 97.5th percentile the upper bound of a 95% confidence interval.

If you have 200 replications that lower bound will be based on the 5th smallest estimate and the highest on the 5th highest. That will be way too small for my taste. My default is 20,000 replications, so the lower bound would be based on the 500th smallest estimate and the upper bound on the 500th largest estimate. The default is nothing but a starting point, and I will often choose another number depending on the exact circumstances.

• Thanks for your answer! But what is the advantage of the resampling? Why can't I just take the 2.5th and 97.5th percentile from my original data? So, when it comes to deciding what number of replications is appropriate, you recommend selecting the largest possible value (if problems with performance are ignored)? – Marcel Grimmer Jul 17 '18 at 11:59
• @MarcelGrimmer: you can bootstrap the confidence interval for a statistic of your data (say, mean, median, variance, or some number that is far more complicated to compute, e.g. generalized prediction error of a model fit to your data). You can report percentiles only for your data itself. Also note that saying, the 2.5th percentile of x is in this confidence interval is not the same as saying the 2.5th percentile of our observations of x is... – cbeleites unhappy with SX Jul 17 '18 at 12:13
• @cbeleites Thats right - I was wrong. Thanks! – Marcel Grimmer Jul 18 '18 at 13:17
• If you use a normal approximation on your bootstrap estimates you can get away doing much less bootstrap replicates though (e.g. 100) than when you use the percentile method (in which case >10 000 is recommended). – Tom Wenseleers Jun 5 '19 at 21:39
• @TomWenseleers true, but you often use the bootstrap because the normal approximation does not apply.... – Maarten Buis Jun 6 '19 at 7:22

According to Efron (the "inventor" of the boostrap technique), you should make 1600 replicas. I have no other clue about where this number comes from, except that its square root is 40, an easy number to divide by. I suggest you go like in any other Monte-Carlo. Try 1600, then increase the bootstrap samples until it stabilizes.

The bootstrap was introduced to compute confidence intervals in case the distribution of the v.a.r is unknown or not technically computable, because of outliers or skew. The bootstrap replaces the theoretical computations of the confidence interval by a measure of simulated samples. So the confidence intervals should be the same.

Note however that all statistical indicators are not equal in front of he boostrap. A average (or a mean) will require less samples than a maximum, or a 1% percentile for example.

• I think you have a typo: it’s not Elon, but Bradley Efron who is the inventor of bootstrap or at least he is the one who popularized bootstraping. – forecaster Jul 17 '18 at 12:29