Are confidence intervals useful?

Question

In frequentist statistics, a 95% confidence interval is an interval-producing procedure that, if repeated an infinite number of times, would contain the true parameter 95% of the time. Why is this useful?

Confidence intervals are often misunderstood. They are not an interval that we can be 95% certain the parameter is in (unless you are using the similar Bayesian credibility interval). Confidence intervals feel like a bait-and-switch to me.

The one use case I can think of is to provide the range of values for which we could not reject the null hypothesis that the parameter is that value. Wouldn't p-values provide this information, but better? Without being so misleading?

In short: Why do we need confidence intervals? How are they, when correctly interpreted, useful?

Answer 1

So long as the confidence interval is treated as random (i.e., looked at from the perspective of treating the data as a set of random variables that we have not seen yet) then we can indeed make useful probability statements about it. Specifically, suppose you have a confidence interval at level $1-\alpha$ for the parameter $\theta$, and the interval has bounds $L(\mathbf{x}) \leqslant U(\mathbf{x})$. Then we can say that:

$$\mathbb{P}(L(\mathbf{X}) \leqslant \theta \leqslant U(\mathbf{X}) | \theta) = 1-\alpha \quad \quad \quad \text{for all } \theta \in \Theta.$$

Moving outside the frequentist paradigm and marginalising over $\theta$ for any prior distribution gives the corresponding (weaker) marginal probability result:

$$\mathbb{P}(L(\mathbf{X}) \leqslant \theta \leqslant U(\mathbf{X})) = 1-\alpha.$$

Once we fix the bounds of the confidence interval by fixing the data to $\mathbf{X} = \mathbb{x}$, we no longer appeal to this probability statement, because we now have fixed the data. However, if the confidence interval is treated as a random interval then we can indeed make this probability statement --- i.e., with probability $1-\alpha$ the parameter $\theta$ will fall within the (random) interval.

Within frequentist statistics, probability statements are statements about relative frequencies over infinitely repeated trials. But that is true of every probability statement in the frequentist paradigm, so if your objection is to relative frequency statements, that is not an objection that is specific to confidence intervals. If we move outside the frequentist paradigm then we can legitimately say that a confidence interval contains its target parameter with the desired probability, so long as we make this probability statement marginally (i.e., not conditional on the data) and we thus treat the confidence interval in its random sense.

I don't know about others, but that seems to me to be a pretty powerful probability result, and a reasonable justification for this form of interval. I am more partial to Bayesian methods myself, but the probability results backing confidence intervals (in their random sense) are powerful results that are not to be sniffed at. Moreover, even within the context of Bayesian analysis, where we let $\theta$ be a random variable with a prior distribution, we can see that the prior predictive probability that the confidence interval contains the parameter is equal to the confidence level. Thus, even within this alternative paradigm, the confidence interval can be regarded as an estimator that has powerful a priori prediction properties.

Answer 2

I agree with @Ben above, and I thought I would provide a simple example of where a Bayesian versus a Frequentist interval would be of value in the same circumstance.

Imagine a factory with parallel assembly lines. It is costly to stop a line, and at the same time, they want to produce quality products. They are concerned about both false positives and false negatives over time. To the factory, it is an averaging process: both power and guaranteed protection against false positives matter. Confidence intervals, as well as tolerance intervals, matter to the factory. Nonetheless, machines will go out of alignment, that is $\theta\ne\Theta$, and detection gear will observe spurious events. The average outcome matters while the specific outcome is an operational detail.

On the opposite side of this is a single customer purchasing a single product or a single lot of products. They do not care about the repetition properties of the assembly line. They care about the one product that they purchased. Let us imagine the customer is NASA and they need the product to meet a specification, say $\gamma\le\Gamma.$ They do not care about the quality of the parts they did not purchase. They need a Bayesian interval of some form. Furthermore, a single failure could kill many astronauts and cost billions of dollars. They need to know that every single part purchased meets specifications. Averaging would be deadly. For a Saturn V rocket, a one percent defect rate would have implied 10,000 defective parts during the Apollo flights. They required 0% defects on all missions.

You worry about having a confidence interval when you are working in the sample space as a factory is doing. It is creating the sample space. You worry about credible intervals when you are working in the parameter space, as a customer would be doing. If you do not care about the observations outside yours, then you are Bayesian. If you do care about the samples that were not seen, but could have been seen, then you are a Frequentist.

Are you concerned with long-run averaging or the specific event?

Answer 3

Note that by the strict definition of confidence interval, it is possible that they are completely meaningless, i.e., not informative about the parameter of interest. However, in practice, they are generally very meaningful.

As an example of a meaningless confidence interval, suppose I have a procedure that 95% of the time produces $[0,1]$, and 5% of the time produces [$U_{min}$, $U_{max}$], where $U_{min}, U_{max}$ are any pair of random variables such that $U_{min} < U_{max}$. Then this is a procedure that captures any probability at least 95% of the time, so is technically a valid confidence interval for any probability. Yet if I told that the interval produced by this procedure was $[0.01, 0.011]$ for a given $p$, you should realize that you really have learned nothing about $p$.

On the other hand, most confidence intervals are constructed in a more useful fashion. For example, if I told you it was created using a Wald Interval procedure, then we know that

$\hat p \text{ } \dot\sim \text{ } N(p, s_e)$

where $s_e$ is the standard error. This is a very meaningful statement about how $\hat p$ relates to $p$. Turning this into a confidence interval is simply an attempt to simplify this result to someone who is not so familiar with normal distributions. That's also not just to say that it's only a tool for people who don't know about normal distributions; for example, the percentile bootstrap is a tool for summarizing the error between the estimator and true parameter when the distribution of this error may be non-Gaussian.

Answer 4

Confidence intervals are not only useful, but essential in some field, such as physics. Unfortunately, the most noise regarding CIs comes from Bayesians caught up in fake debates with Frequentists, usually in the context of social "sciences" and other science-like disciplines.

Suppose that I measure a quantity in Physics, such as electricity charge. I would always supply it with the measure of uncertainty of the value, which is usually a standard deviation. Since, in Physics errors are often Gaussian, this directly translated into CI. However, when the errors are not Gaussian, it gets a bit complicated, some integrals need to be evaluated etc. Nothing too esoteric though usually.

Here's a brief presentation on the CI in particle physics, and the definition:

quantitative statement about the fraction of times that such an interval would contain the true value of the parameter in a large number of repeated experiments

Note, that in Physics "repeated experiments" has often a literal meaning: it's assumed you can actually repeat experiments in the paper, and would actually observe that fraction. So, the CI has almost a literal meaning to you, and is just a way to express the information about the uncertainty of the measurement. It's not a thought experiment, not a subjective opinion, not your or my feelings about likelihoods etc. It's what you were able to devise from experiments, and what I should be able to observe when reproducing your experiment.

Answer 5

This thread has devolved quickly into the Frequentist vs Bayesian debate, and that is not easily resolvable. The math in both approaches is solid, so it always comes down to philosophical preferences. The frequentist interpretation of probability as the limit of an event's relative frequency is justified by the strong law of large numbers; regardless of your preferred interpretation of probability, an event's relative frequency will converge to its probability with probability 1.

Frequentist confidence intervals are indeed trickier to interpret than Bayesian credible intervals. By treating an unknown quantity as a random variable, Bayesians can assert that one interval contains that quantity with some probability. Frequentists refuse to treat some quantities as random variables, and any equations containing only constants can only be true or false. So when estimating an unknown constant, frequentists must bound them with a RANDOM interval to involve probability at all. Rather than one interval containing a random variable with some probability, a frequentist method generates many different possible intervals, some of which contains the unknown constant. If the coverage probability is reasonably high, it's a reasonable leap of faith to assert that a particular interval contains the unknown constant (note, not "with some probability").

A Bayesian would balk at such a leap of faith as much as a Frequentist balks at treating any unknown quantity as a random variable. The frequentist Neyman construction method in fact exposed an embarrassing issue with such leaps of faith. Without actively preventing it (see Feldman and Cousins, 1997 for one approach), rare outcomes may generate EMPTY confidence intervals for a distribution parameter. Such a leap of faith would be very unreasonable! I've seen a few Bayesians using that example to mock frequentist methods, while frequentists typically respond with "well I still get a correct interval most of the time, and without making false assumptions." I'll point out that the Bayesian/frequentist impasse is not important to most who apply their methods. Even people who are committed to coverage probability will often use Bayesian methods if the methods are shown to have good coverage probability in simulations.