I've come across the phrase "honest confidence interval" but I have never seen a definition. What is it?
9
-
1$\begingroup$ I haven't heard of it before but a quick search leads to: jstor.org/stable/2241707 $\endgroup$– Sextus EmpiricusCommented Oct 8, 2022 at 13:00
-
1$\begingroup$ In what context did you read it? I can speculate (I think correctly) that it means a confidence interval at the $(1-\alpha)\times 100\%$-level will contain the true parameter in the claimed percentage of replications, but I’ve never heard this term before. $\endgroup$– DaveCommented Oct 8, 2022 at 13:02
-
1$\begingroup$ I might claim a confidence of $95\%$, but who is to say that the procedure really would cover the true parameter in $95\%$ of replications? $\endgroup$– DaveCommented Oct 8, 2022 at 13:21
-
1$\begingroup$ A commonly occuring example of "some other estimate" in that case might be "the mean and standard deviation after removing seeming outliers" or "the mean and variance after looking at the data to see if some transformation might be needed". The usual pivotal quantity used to derive the interval is no longer distributed as t. $\endgroup$– Glen_bCommented Oct 9, 2022 at 10:02
-
2$\begingroup$ @SextusEmpiricus the project euclid link offers a free-access pdf for IMS publications (and others; it hosts over 100 journals); the paper is available here: projecteuclid.org/journals/annals-of-statistics/volume-17/… $\endgroup$– Glen_bCommented Oct 9, 2022 at 10:12
|
Show 4 more comments
1 Answer
$\begingroup$
$\endgroup$
7
An example of an article that defines it is "Honest Confidence Regions for Nonparametric Regression" by Ker-Chau Li (1989).
Here the word "honest" refers to the requirement that the minimum coverage probability over a rich class of (nonparametric) regression functions should be no less than the nominal confidence level
Confidence intervals can 'fail' to be honest when they are not exact computations, and are instead estimates.
An example of how it can go 'wrong' are the different methods to compute a confidence intervals for a binomial proportion. The Clopper-Pearson is always covering at least the given percentage, no matter what the true proportion parameter is. Other intervals, like Wald interval or Jeffreys interval don't (at least not when you condition on the parameter).
The image below gives an example where the coverage probability is given as function of the true parameter for the case of a sample from a binomial distribution with n=100 (From this question Revisiting the Rule of Three)
In the case of the Wald interval the 'mistakes' are introduced because it is an approximation. In the case of Jeffreys' interval the 'mistake' is because the interval is actually a credible interval and doesn't even try to be a confidence interval by design.
answered Oct 8, 2022 at 14:12
-
2$\begingroup$ That's really interesting; I always thought that achieving a certain coverage rate was actually the definition of a confidence interval tout court. $\endgroup$ Commented Oct 8, 2022 at 14:30
-
3$\begingroup$ @JohnMadden that definition is still correct. It is only that what we often use as confidence intervals are in fact approximations of confidence intervals. $\endgroup$ Commented Oct 8, 2022 at 15:06
-
-
$\begingroup$ The extract in the answer appears to use the word 'honest' to signify a type of robustness of CI coverage across models. I think we are fortunate that the use of the word honest in this way never caught on. $\endgroup$ Commented Oct 8, 2022 at 22:38
-
1$\begingroup$ @SextusEmpiricus That works. $\endgroup$ Commented Oct 9, 2022 at 1:37