Today I am faced with the term bootstrap confidence interval coverage. Unfortunately, I did not find any simple and clear explanation of the term. Can you help me please?
$\begingroup$
$\endgroup$
3
-
4$\begingroup$ Do you know what it means for a "regular" confidence interval? $\endgroup$– DaveCommented Jul 1, 2020 at 16:22
-
2$\begingroup$ See stats.stackexchange.com/search?q=confidence+interval+coverage. For answers focused on bootstrap CIs see stats.stackexchange.com/…. $\endgroup$– whuber ♦Commented Jul 1, 2020 at 16:48
-
$\begingroup$ @Dave Yes, sure. I'm familiar both with bootstrapping and CI terms, but know nothing about coverage. $\endgroup$– DenisCommented Jul 1, 2020 at 17:14
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
The term, 'bootstrap confidence interval coverage' is the combination of three concepts: 1. bootstrapping 2. confidence interval 3. coverage probability
bootstrapping: The bootstrapping is the resampling method to calculate a statistic. It draws $N$ sets of samples randomly from the original sample with replacement and calculates $N$ statistic from the each of the data sets.
confidence interval(CI): The CI is the interval where the calculated statistic for the specific data sample would lay inside with a given probability.
coverage probability: The coverage probability is the given probability we require. The calculated statistic from the bootstrapping would be inside the CI with the coverage probability.
Summing up, the CI of the statistic is calculated from the bootstrapped samples. And the probability with which the statistic for a given data sample would be inside the CI is the quantity described by the 'bootstrap confidence interval coverage'.
answered Jul 1, 2020 at 17:03
-
$\begingroup$ Thanks a lot. From your explanation it seems that
bootstrap confidence interval coverageis the same asconfidence level. Am i right? E.g. if one concludes "there is a 95% probability that the true value may lie in range of 5 to 10." Can i say thatbootstrap confidence interval coveragein that case would be 95%? $\endgroup$– DenisCommented Jul 1, 2020 at 17:31 -
1$\begingroup$ CI is the interval and coverage is the corresponding probability for the CI. And yes, in the case, 95% is the coverage. $\endgroup$ Commented Jul 1, 2020 at 17:37
-
1$\begingroup$ I think it might be important to separate nominal or target coverage (e.g. 95% for a 95% confidence interval) and the actual coverage for a specific bootstrap confidence interval. I see this answer as incomplete because it does not address the fact that so-called 95% CI based on bootstrap methods may in fact be wrong (not achieve 95% coverage). $\endgroup$ Commented Jul 6, 2020 at 13:41
$\begingroup$
$\endgroup$
This answer takes the same framework as @kevin012's answer, but I'd like to try to be more precise in some of the definitions.
I have no trouble with the definition of the bootstrap, or resampling data with replacement and computing the statistic from the Monte Carlo samples.
a) Let's say there's as confidence level, $1-\alpha$, and that this is our "nominal coverage level" as well. Now
b) a $1-\alpha * 100\%$ confidence interval is, in theory, an interval that contains the test statistic computed from $1-\alpha * 100\%$ of random samples from the population (i.e. there is an $\alpha * 100\%$ chance of drawing a random sample from the population with a test statistic outside the confidence interval). The key here is that lots of CI don't actually work this way; because it's hard to get that right without pivotal quantities and continuous statistics.
c) There are lots of bootstrap confidence intervals. Below is an example with the Basic Bootstrap CI as implemented in the R package
bootStatistical coverage is the [expected] frequency that a so-called "confidence interval" actually contains its target value. It is not the case that various bootstrap CI always work as advertised.
Below is some R code showing that the coverage of the 95% basic bootstrap CI is woefully low for 50 Bernoulli trials with $p=0.1$. It is around 86%, when of course it should be 95% by definition.
library(boot)
library(tidyverse)
n<-50 #sample size
p<-0.1 #probability of success
set.seed(1615) #set random seed for replicability
my_p<-function(x, indices){sum(x[indices])/length(x[indices])} #define test statistic for boot
test_coverage<-map_dfr(1:4000, function(x){ # repeatedly bootstrap to determine empirical coverage
empirical<-rbinom(n, 1, p) #take empirical sample
ci<-boot.ci(boot(data = empirical, statistic = my_p, R = n)
, conf = 0.95
, type = "basic")
return(data.frame(lwr=ci$basic[4], upr=ci$basic[5]))
})
outside<- function(p, ci){ #fraction of samples for which true parameter falls outside CI
(sum(p<ci$lwr)+sum(p>ci$upr))/length(ci$lwr)
}
cov_obs<-1-outside(p, test_coverage)
cov_obs # 0.86, should be 0.95
