One proves mathematically that if assumptions of a model are satisfied, then the coverage rate of a $100p\%$ confidence interval is $100p\%$. But then statistics gets applied to the world, where model assumptions may not be satisfied. Are there any studies comparing the coverage rates of confidence intervals applied to the real world with theoretical coverage rates?
$\begingroup$
$\endgroup$
5
asked Oct 11, 2013 at 17:31
-
2$\begingroup$ Comparisons to the "real world" are next to impossible except in contrived situations: after all, we compute the CIs precisely because we do not know the values they target. However, there are a huge number of simulation studies of coverage of CIs: it is practically unimaginable that there exists any CI in existence that has not been studied in that way. Having said that, there are some notable exceptions, such as a study of the CIs for the speed of light provided by physicists. $\endgroup$– whuber ♦Commented Oct 11, 2013 at 18:06
-
$\begingroup$ We do not know the true values at the time we form confidence intervals, but in some cases one learns them later. $\endgroup$– Michael HardyCommented Oct 11, 2013 at 19:49
-
1$\begingroup$ Yes; that is the case with the speed of light experiments, which cover 90 years and ultimately are compared to a consensus value obtained 25 years after that. But it took a long time to pin down even this fundamental physical constant. In other fields (economics, for instance), finding true values typically is impossible, yet their estimates likely are subject to much more unexpected and unmodeled error. We should be cautious in generalizing CI coverage results from one field to another or even from one kind of experiment to another within a field. $\endgroup$– whuber ♦Commented Oct 11, 2013 at 20:00
-
2$\begingroup$ One major problem is the issue of biased intervals (e.g. caused by a model that misses a possibly small but important effect). Perhaps counterintuitively (until you understand what's happening, at least), in real world problems coverage tends to get worse as sample sizes increase. For example, a 90% interval might have 88% actual coverage at $n=20$ and say 25% actual coverage at $n=10000$... and one that keeps decreasing with larger $n$. $\endgroup$– Glen_bCommented Oct 12, 2013 at 0:20
-
$\begingroup$ @whuber ..... you say "except in contrived situations". There's another term for "contrived situations": experiments. Has anyone thought about what sorts of experiments could address this? $\endgroup$– Michael HardyCommented Aug 15, 2014 at 21:52
Add a comment
|