Is there a useful way to define the “best” confidence interval?

The standard definition of (say) a 95% confidence interval (CI) simply requires that the probability that it contains the true parameter is 95%. Obviously, this is not unique. The language I've seen suggests that among the many valid CI, it usually makes sense to find something like the shortest, or symmetric, or known precisely even when some distribution parameters are unknown, etc. In other words, there seems to be no obvious hierarchy of what CI are "better" than others.

However, I thought one equivalent definition of CI is that it consists of all values such that the null hypothesis that the true parameter equals that value wouldn't be rejected at the appropriate significance level after seeing the realized sample. This suggests that as long as we choose a test that we like, we can automatically construct the CI. And there's a standard preference among tests based on the concept of UMP (or UMP among unbiased tests).

Is there any benefit in defining CI as the one corresponding to the UMP test or something like that?

• (+1) There are two difficulties with your proposal. The first is that UMP tests often don't exist. The second is that a fruitful way to evaluate confidence intervals is in terms of a related loss function. This allows for a flexible but wide variety of solutions. – whuber Mar 2 '16 at 20:39
• The definition of a confidence interval is not what you claim, as discussed in several places on this site, e.g. here: stats.stackexchange.com/questions/13655/… and here: stats.stackexchange.com/a/6431/5829 – Jake Westfall Mar 2 '16 at 21:14
• What is the purpose of CI for you? What are you trying to do with it? – Aksakal Mar 2 '16 at 22:19
• @Aksakal actually, I just realized that I'm very confused about the concept and wanted to understand it better. Not specific to any application. – max Mar 2 '16 at 23:01
• @JakeWestfall I found an interesting discussion but no clear definition in either of those links. Can you point me to a specific definition? I didn't claim my definition was correct, but that's all I found after a brief search of statistics textbooks. In fact, the same definition is offered in the Morey paper mentioned in Alex's answer – max Mar 3 '16 at 1:02

A bit long for a comment. Check out the discussion on UMP's in this paper "The fallacy of placing confidence in confidence intervals" by Morey et al. In particular, there are some examples where:

"Even more strangely, intervals from the UMP procedure initially increase in width with the uncertainty in the data, but when the width of the likelihood is greater than 5 meters, the width of the UMP interval is inversely related to the uncertainty in the data, like the nonparametric interval. The UMP and sampling distribution procedures share the dubious distinction that their CIs cannot be used to work backwards to the observations. In spite of being the “most powerful” procedure, the UMP procedure clearly throws away important information."

• What about the other direction? Given a good confidence interval, is the corresponding null hypothesis test generally good? – max Apr 16 '16 at 3:15

Rejection is only part of the inference, don't get stuck there. You're making a decision. Let's say you need to decide whether to go to a mechanic when "check engine" light goes on or forget about it.

So, your null hypothesis is that the engine's fine, and the light is just the nuisance. The check engine light is your test. Let's say the p-value is 5%, while your significance is $\alpha=0.01$, so you can't reject the null, and move on with your business. This is how statistical significance works in its naive form.

That's not how decisions to be made and how economic significance should be taken into account. You have to calculate the cost of going with null vs. rejecting it and selecting the alternative hypo.

I completely omitted the alternative hypothesis in above example, because that's how everybody does it: they think alternative hypo is just some kind of formality like curtsy. In real life alternative is as important as null, because that's how you calculate the cost of not choosing the null. Only when you account for costs of null and alternative, you should be making the decision to go or not to go to a mechanic. The p-value and confidence intervals on theirs own have no meaning in this regard, only in conjuction with costs they become meaningful

• I don't mean that rejection is the end of the story. I was just really surprised when I realized all the definitions of CI I found allowed non-unique CI. With all the other caveats around using CI, I hoped to at least not have to deal with the fact that the choice of any particular CI is arbitrary. – max Mar 3 '16 at 6:40