1
$\begingroup$

Does it make much sense to calculate confidence intervals for coverage probabilities?

For instance, if a CI coverage probability is estimated to be 0.80, does computing an exact 95% binomial confidence interval for the "true" coverage of (0.6823120, 0.8889842) (actually computed in R via binomial.test()) mean anything? That is, is such an interval "useful"?

$\endgroup$

1 Answer 1

1
$\begingroup$

The coverage of a CI is a binomial random variable, so yea I think it would make sense to have uncertainty estimates reported in a simulation study.

That being said, I if the CI is cheap to compute I think we can likely just run sufficiently many simulations so that the resulting simulation error is within some desirable tolerance.

I would probably only care about the first two digits of precision in a coverage probability, meaning if you ran 10,000 simulations the uncertainty estimates likely wouldn't add much.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.