What is the coverage of bootstrap confidence interval? Today I am faced with the term bootstrap confidence interval coverage. Unfortunately, I did not find any simple and clear explanation of the term. Can you help me please?
 A: The term, 'bootstrap confidence interval coverage' is the combination of three concepts: 1. bootstrapping 2. confidence interval 3. coverage probability

*

*bootstrapping: The bootstrapping is the resampling method to calculate a statistic. It draws $N$ sets of samples randomly from the original sample with replacement and calculates $N$ statistic from the each of the data sets.


*confidence interval(CI): The CI is the interval where the calculated statistic for the specific data sample would lay inside with a given probability.


*coverage probability: The coverage probability is the given probability we require. The calculated statistic from the bootstrapping would be inside the CI with the coverage probability.
Summing up, the CI of the statistic is calculated from the bootstrapped samples. And the probability with which the statistic for a given data sample would be inside the CI is the quantity described by the 'bootstrap confidence interval coverage'.
A: This answer takes the same framework as @kevin012's answer, but I'd like to try to be more precise in some of the definitions.

*

*I have no trouble with the definition of the bootstrap, or resampling data with replacement and computing the statistic from the Monte Carlo samples.


*a) Let's say there's as confidence level, $1-\alpha$, and that this is our "nominal coverage level" as well. Now
b) a $1-\alpha * 100\%$ confidence interval is, in theory, an interval that contains the test statistic computed from $1-\alpha * 100\%$ of random samples from the population (i.e. there is an $\alpha * 100\%$ chance of drawing a random sample from the population with a test statistic outside the confidence interval). The key here is that lots of CI don't actually work this way; because it's hard to get that right without pivotal quantities and continuous statistics.
c) There are lots of bootstrap confidence intervals. Below is an example with the Basic Bootstrap CI as implemented in the R package boot


*Statistical coverage is the [expected] frequency that a so-called "confidence interval" actually contains its target value. It is not the case that various bootstrap CI always work as advertised.
Below is some R code showing that the coverage of the 95% basic bootstrap CI is woefully low for 50 Bernoulli trials with $p=0.1$. It is around 86%, when of course it should be 95% by definition.
 library(boot)
 library(tidyverse)

 n<-50 #sample size
 p<-0.1 #probability of success

  set.seed(1615) #set random seed for replicability



 my_p<-function(x, indices){sum(x[indices])/length(x[indices])} #define test statistic for boot


 test_coverage<-map_dfr(1:4000, function(x){     # repeatedly bootstrap to determine empirical coverage
   empirical<-rbinom(n, 1, p) #take empirical sample  
   ci<-boot.ci(boot(data = empirical, statistic = my_p, R = n)
         , conf = 0.95
         , type = "basic")
   return(data.frame(lwr=ci$basic[4], upr=ci$basic[5]))
 })

 outside<- function(p, ci){ #fraction of samples for which true parameter falls outside CI
   (sum(p<ci$lwr)+sum(p>ci$upr))/length(ci$lwr)
 }
 cov_obs<-1-outside(p, test_coverage)

 cov_obs # 0.86, should be 0.95

