# What is the coverage of bootstrap confidence interval?

Today I am faced with the term bootstrap confidence interval coverage. Unfortunately, I did not find any simple and clear explanation of the term. Can you help me please?

## 2 Answers

The term, 'bootstrap confidence interval coverage' is the combination of three concepts: 1. bootstrapping 2. confidence interval 3. coverage probability

1. bootstrapping: The bootstrapping is the resampling method to calculate a statistic. It draws $$N$$ sets of samples randomly from the original sample with replacement and calculates $$N$$ statistic from the each of the data sets.

2. confidence interval(CI): The CI is the interval where the calculated statistic for the specific data sample would lay inside with a given probability.

3. coverage probability: The coverage probability is the given probability we require. The calculated statistic from the bootstrapping would be inside the CI with the coverage probability.

Summing up, the CI of the statistic is calculated from the bootstrapped samples. And the probability with which the statistic for a given data sample would be inside the CI is the quantity described by the 'bootstrap confidence interval coverage'.

• Thanks a lot. From your explanation it seems that bootstrap confidence interval coverage is the same as confidence level. Am i right? E.g. if one concludes "there is a 95% probability that the true value may lie in range of 5 to 10." Can i say that bootstrap confidence interval coverage in that case would be 95%? – Denis Jul 1 '20 at 17:31
• CI is the interval and coverage is the corresponding probability for the CI. And yes, in the case, 95% is the coverage. – hbadger19042 Jul 1 '20 at 17:37
• I think it might be important to separate nominal or target coverage (e.g. 95% for a 95% confidence interval) and the actual coverage for a specific bootstrap confidence interval. I see this answer as incomplete because it does not address the fact that so-called 95% CI based on bootstrap methods may in fact be wrong (not achieve 95% coverage). – Michael Jul 6 '20 at 13:41

This answer takes the same framework as @kevin012's answer, but I'd like to try to be more precise in some of the definitions.

1. I have no trouble with the definition of the bootstrap, or resampling data with replacement and computing the statistic from the Monte Carlo samples.

2. a) Let's say there's as confidence level, $$1-\alpha$$, and that this is our "nominal coverage level" as well. Now

b) a $$1-\alpha * 100\%$$ confidence interval is, in theory, an interval that contains the test statistic computed from $$1-\alpha * 100\%$$ of random samples from the population (i.e. there is an $$\alpha * 100\%$$ chance of drawing a random sample from the population with a test statistic outside the confidence interval). The key here is that lots of CI don't actually work this way; because it's hard to get that right without pivotal quantities and continuous statistics.

c) There are lots of bootstrap confidence intervals. Below is an example with the Basic Bootstrap CI as implemented in the R package boot

3. Statistical coverage is the [expected] frequency that a so-called "confidence interval" actually contains its target value. It is not the case that various bootstrap CI always work as advertised.

Below is some R code showing that the coverage of the 95% basic bootstrap CI is woefully low for 50 Bernoulli trials with $$p=0.1$$. It is around 86%, when of course it should be 95% by definition.

 library(boot)
library(tidyverse)

n<-50 #sample size
p<-0.1 #probability of success

set.seed(1615) #set random seed for replicability

my_p<-function(x, indices){sum(x[indices])/length(x[indices])} #define test statistic for boot

test_coverage<-map_dfr(1:4000, function(x){     # repeatedly bootstrap to determine empirical coverage
empirical<-rbinom(n, 1, p) #take empirical sample
ci<-boot.ci(boot(data = empirical, statistic = my_p, R = n)
, conf = 0.95
, type = "basic")
return(data.frame(lwr=ci$$basic, upr=ci$$basic))
})

outside<- function(p, ci){ #fraction of samples for which true parameter falls outside CI
(sum(p<ci$$lwr)+sum(p>ci$$upr))/length(ci\$lwr)
}
cov_obs<-1-outside(p, test_coverage)

cov_obs # 0.86, should be 0.95