Is there a useful way to define the "best" confidence interval?

Question

The standard definition of (say) a 95% confidence interval (CI) simply requires that the probability that it contains the true parameter is 95%. Obviously, this is not unique. The language I've seen suggests that among the many valid CI, it usually makes sense to find something like the shortest, or symmetric, or known precisely even when some distribution parameters are unknown, etc. In other words, there seems to be no obvious hierarchy of what CI are "better" than others.

However, I thought one equivalent definition of CI is that it consists of all values such that the null hypothesis that the true parameter equals that value wouldn't be rejected at the appropriate significance level after seeing the realized sample. This suggests that as long as we choose a test that we like, we can automatically construct the CI. And there's a standard preference among tests based on the concept of UMP (or UMP among unbiased tests).

Is there any benefit in defining CI as the one corresponding to the UMP test or something like that?

Answer 1

A bit long for a comment. Check out the discussion on UMP's in this paper "The fallacy of placing confidence in confidence intervals" by Morey et al. In particular, there are some examples where:

"Even more strangely, intervals from the UMP procedure initially increase in width with the uncertainty in the data, but when the width of the likelihood is greater than 5 meters, the width of the UMP interval is inversely related to the uncertainty in the data, like the nonparametric interval. The UMP and sampling distribution procedures share the dubious distinction that their CIs cannot be used to work backwards to the observations. In spite of being the “most powerful” procedure, the UMP procedure clearly throws away important information."

Answer 2

Rejection is only part of the inference, don't get stuck there. You're making a decision. Let's say you need to decide whether to go to a mechanic when "check engine" light goes on or forget about it.

So, your null hypothesis is that the engine's fine, and the light is just the nuisance. The check engine light is your test. Let's say the p-value is 5%, while your significance is $\alpha=0.01$, so you can't reject the null, and move on with your business. This is how statistical significance works in its naive form.

That's not how decisions to be made and how economic significance should be taken into account. You have to calculate the cost of going with null vs. rejecting it and selecting the alternative hypo.

I completely omitted the alternative hypothesis in above example, because that's how everybody does it: they think alternative hypo is just some kind of formality like curtsy. In real life alternative is as important as null, because that's how you calculate the cost of not choosing the null. Only when you account for costs of null and alternative, you should be making the decision to go or not to go to a mechanic. The p-value and confidence intervals on theirs own have no meaning in this regard, only in conjuction with costs they become meaningful