One proves mathematically that if assumptions of a model are satisfied, then the coverage rate of a $100p\%$ confidence interval is $100p\%$. But then statistics gets applied to the world, where model assumptions may not be satisfied. Are there any studies comparing the coverage rates of confidence intervals applied to the real world with theoretical coverage rates?

  • 2
    $\begingroup$ Comparisons to the "real world" are next to impossible except in contrived situations: after all, we compute the CIs precisely because we do not know the values they target. However, there are a huge number of simulation studies of coverage of CIs: it is practically unimaginable that there exists any CI in existence that has not been studied in that way. Having said that, there are some notable exceptions, such as a study of the CIs for the speed of light provided by physicists. $\endgroup$ – whuber Oct 11 '13 at 18:06
  • $\begingroup$ We do not know the true values at the time we form confidence intervals, but in some cases one learns them later. $\endgroup$ – Michael Hardy Oct 11 '13 at 19:49
  • 1
    $\begingroup$ Yes; that is the case with the speed of light experiments, which cover 90 years and ultimately are compared to a consensus value obtained 25 years after that. But it took a long time to pin down even this fundamental physical constant. In other fields (economics, for instance), finding true values typically is impossible, yet their estimates likely are subject to much more unexpected and unmodeled error. We should be cautious in generalizing CI coverage results from one field to another or even from one kind of experiment to another within a field. $\endgroup$ – whuber Oct 11 '13 at 20:00
  • 2
    $\begingroup$ One major problem is the issue of biased intervals (e.g. caused by a model that misses a possibly small but important effect). Perhaps counterintuitively (until you understand what's happening, at least), in real world problems coverage tends to get worse as sample sizes increase. For example, a 90% interval might have 88% actual coverage at $n=20$ and say 25% actual coverage at $n=10000$... and one that keeps decreasing with larger $n$. $\endgroup$ – Glen_b Oct 12 '13 at 0:20
  • $\begingroup$ @whuber ..... you say "except in contrived situations". There's another term for "contrived situations": experiments. Has anyone thought about what sorts of experiments could address this? $\endgroup$ – Michael Hardy Aug 15 '14 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.