Are confidence intervals useful?

In frequentist statistics, a 95% confidence interval is an interval-producing procedure that, if repeated an infinite number of times, would contain the true parameter 95% of the time. Why is this useful?

Confidence intervals are often misunderstood. They are not an interval that we can be 95% certain the parameter is in (unless you are using the similar Bayesian credibility interval). Confidence intervals feel like a bait-and-switch to me.

The one use case I can think of is to provide the range of values for which we could not reject the null hypothesis that the parameter is that value. Wouldn't p-values provide this information, but better? Without being so misleading?

In short: Why do we need confidence intervals? How are they, when correctly interpreted, useful?

• – Nat Jan 31 '19 at 11:47
• The Bayesian credibility interval is neither an interval that we can be 95% certain the parameter is in. – Sextus Empiricus Feb 1 '19 at 12:32
• @MartijnWeterings: unless you are 100% certain of your prior. – Xi'an Feb 4 '19 at 6:55
• @Xi'an that works when a parameter $\theta$ is 100% certain to be reasonably considered a random variable and an experiment is like sampling from a joint frequency distribution $P(\theta,x)$, ie you use Bayes rule as: $P(\theta|x) = P(\theta,x)/P(x)$ without explicit 'prior'. It's not the same for a parameter that is considered to be fixed. Then the posterior beliefs would require you to also 'update' the old joint frequency distribution of $X$ and $\theta$. It is a bit absurd to claim to be updating 'prior beliefs' which were 100% sure. – Sextus Empiricus Feb 4 '19 at 7:27

5 Answers

So long as the confidence interval is treated as random (i.e., looked at from the perspective of treating the data as a set of random variables that we have not seen yet) then we can indeed make useful probability statements about it. Specifically, suppose you have a confidence interval at level $$1-\alpha$$ for the parameter $$\theta$$, and the interval has bounds $$L(\mathbf{x}) \leqslant U(\mathbf{x})$$. Then we can say that:

$$\mathbb{P}(L(\mathbf{X}) \leqslant \theta \leqslant U(\mathbf{X}) | \theta) = 1-\alpha \quad \quad \quad \text{for all } \theta \in \Theta.$$

Moving outside the frequentist paradigm and marginalising over $$\theta$$ for any prior distribution gives the corresponding (weaker) marginal probability result:

$$\mathbb{P}(L(\mathbf{X}) \leqslant \theta \leqslant U(\mathbf{X})) = 1-\alpha.$$

Once we fix the bounds of the confidence interval by fixing the data to $$\mathbf{X} = \mathbb{x}$$, we no longer appeal to this probability statement, because we now have fixed the data. However, if the confidence interval is treated as a random interval then we can indeed make this probability statement --- i.e., with probability $$1-\alpha$$ the parameter $$\theta$$ will fall within the (random) interval.

Within frequentist statistics, probability statements are statements about relative frequencies over infinitely repeated trials. But that is true of every probability statement in the frequentist paradigm, so if your objection is to relative frequency statements, that is not an objection that is specific to confidence intervals. If we move outside the frequentist paradigm then we can legitimately say that a confidence interval contains its target parameter with the desired probability, so long as we make this probability statement marginally (i.e., not conditional on the data) and we thus treat the confidence interval in its random sense.

I don't know about others, but that seems to me to be a pretty powerful probability result, and a reasonable justification for this form of interval. I am more partial to Bayesian methods myself, but the probability results backing confidence intervals (in their random sense) are powerful results that are not to be sniffed at. Moreover, even within the context of Bayesian analysis, where we let $$\theta$$ be a random variable with a prior distribution, we can see that the prior predictive probability that the confidence interval contains the parameter is equal to the confidence level. Thus, even within this alternative paradigm, the confidence interval can be regarded as an estimator that has powerful a priori prediction properties.

• "Moving outside the frequentist paradigm" isn't that exactly the problem? In general we want an interval that contains the true value of a parameter of interest with some probability. No frequentist analysis can give us that, and implicitly re-interpreting it as a Bayesian analysis leads to misunderstandings. Better to answer the question directly via a Bayesian credible interval. There are uses for confidence intervals where you are repeatedly performing "experiments", e.g. quality control. – Dikran Marsupial Jan 31 '19 at 8:18
• It is not a matter of implicitly reinterpreting as Bayesian (the latter would condition on the data to get a posterior). The answer is merely showing the OP that we can make useful probability statements about the confidence interval. As to more general objections to the frequentist paradigm, those are well and good, but they are not objections specific to confidence intervals. – Ben - Reinstate Monica Jan 31 '19 at 8:57
• As you can see from the above probability statements, we can guarantee that the CI contains the parameter with some probability, so long as we look at this a priori. – Ben - Reinstate Monica Jan 31 '19 at 9:09
• If you have moved out of the frequentist paradigm, but are not moving to a Bayesian framework, what framework is it? I wasn't expressing an objection to frequentism, I believe you should use the framework that most directly answers the question you actually want to pose. Confidence and credible intervals answer different questions. – Dikran Marsupial Jan 31 '19 at 11:11
• @Dikran: The probability statement stands as written, and is a pure mathematical statement. I really don't see how you can reasonably object to this. – Ben - Reinstate Monica Feb 5 '19 at 8:20

I agree with @Ben above, and I thought I would provide a simple example of where a Bayesian versus a Frequentist interval would be of value in the same circumstance.

Imagine a factory with parallel assembly lines. It is costly to stop a line, and at the same time, they want to produce quality products. They are concerned about both false positives and false negatives over time. To the factory, it is an averaging process: both power and guaranteed protection against false positives matter. Confidence intervals, as well as tolerance intervals, matter to the factory. Nonetheless, machines will go out of alignment, that is $$\theta\ne\Theta$$, and detection gear will observe spurious events. The average outcome matters while the specific outcome is an operational detail.

On the opposite side of this is a single customer purchasing a single product or a single lot of products. They do not care about the repetition properties of the assembly line. They care about the one product that they purchased. Let us imagine the customer is NASA and they need the product to meet a specification, say $$\gamma\le\Gamma.$$ They do not care about the quality of the parts they did not purchase. They need a Bayesian interval of some form. Furthermore, a single failure could kill many astronauts and cost billions of dollars. They need to know that every single part purchased meets specifications. Averaging would be deadly. For a Saturn V rocket, a one percent defect rate would have implied 10,000 defective parts during the Apollo flights. They required 0% defects on all missions.

You worry about having a confidence interval when you are working in the sample space as a factory is doing. It is creating the sample space. You worry about credible intervals when you are working in the parameter space, as a customer would be doing. If you do not care about the observations outside yours, then you are Bayesian. If you do care about the samples that were not seen, but could have been seen, then you are a Frequentist.

Are you concerned with long-run averaging or the specific event?

• Does NASA actually purchase parts based on Bayesian intervals? I understand your point, but do they actually do it? – Aksakal Jan 31 '19 at 17:55
• @Aksakal I do not know. Juran, of course, wrote a wonderful work on quality assurance at NASA, but I cannot remember at all if the testing process was discussed as it has been more than a decade since I read it. I know that W Edwards Deming was opposed to confidence intervals in favor of credible intervals, but again, that doesn't directly pertain. My guess, and I do know people who would know but it is inconvenient to ask at the moment, is that they use Frequentist methods because that is what most people are trained in. You use the hammer you have. – Dave Harris Jan 31 '19 at 17:58
• Is it the case of "a hammer" though? Maybe it has something to do with the way things are in engineering? – Aksakal Jan 31 '19 at 18:09
• @Aksakal I am not qualified to opine on that. – Dave Harris Jan 31 '19 at 19:56
• Say a company makes $n$ parts, with a $\alpha$ level composite hypothesis test $H_0: \gamma > \Gamma$ you have them tested for mistakes: $x$ of them pass without mistakes and $y$ of them fail. You can give NASA a reasonable guarantee. The maximum amount of products that can accidentally pass the test (wrongly considered without error) is $n\alpha$. Knowing that you sold $x$ items you can compute a maximum probability that a sold part is actually not in accordance with the alternative hypothesis $\gamma \leq \Gamma$. – Sextus Empiricus Feb 4 '19 at 9:01

Note that by the strict definition of confidence interval, it is possible that they are completely meaningless, i.e., not informative about the parameter of interest. However, in practice, they are generally very meaningful.

As an example of a meaningless confidence interval, suppose I have a procedure that 95% of the time produces $$[0,1]$$, and 5% of the time produces [$$U_{min}$$, $$U_{max}$$], where $$U_{min}, U_{max}$$ are any pair of random variables such that $$U_{min} < U_{max}$$. Then this is a procedure that captures any probability at least 95% of the time, so is technically a valid confidence interval for any probability. Yet if I told that the interval produced by this procedure was $$[0.01, 0.011]$$ for a given $$p$$, you should realize that you really have learned nothing about $$p$$.

On the other hand, most confidence intervals are constructed in a more useful fashion. For example, if I told you it was created using a Wald Interval procedure, then we know that

$$\hat p \text{ } \dot\sim \text{ } N(p, s_e)$$

where $$s_e$$ is the standard error. This is a very meaningful statement about how $$\hat p$$ relates to $$p$$. Turning this into a confidence interval is simply an attempt to simplify this result to someone who is not so familiar with normal distributions. That's also not just to say that it's only a tool for people who don't know about normal distributions; for example, the percentile bootstrap is a tool for summarizing the error between the estimator and true parameter when the distribution of this error may be non-Gaussian.

Confidence intervals are not only useful, but essential in some field, such as physics. Unfortunately, the most noise regarding CIs comes from Bayesians caught up in fake debates with Frequentists, usually in the context of social "sciences" and other science-like disciplines.

Suppose that I measure a quantity in Physics, such as electricity charge. I would always supply it with the measure of uncertainty of the value, which is usually a standard deviation. Since, in Physics errors are often Gaussian, this directly translated into CI. However, when the errors are not Gaussian, it gets a bit complicated, some integrals need to be evaluated etc. Nothing too esoteric though usually.

Here's a brief presentation on the CI in particle physics, and the definition:

quantitative statement about the fraction of times that such an interval would contain the true value of the parameter in a large number of repeated experiments

Note, that in Physics "repeated experiments" has often a literal meaning: it's assumed you can actually repeat experiments in the paper, and would actually observe that fraction. So, the CI has almost a literal meaning to you, and is just a way to express the information about the uncertainty of the measurement. It's not a thought experiment, not a subjective opinion, not your or my feelings about likelihoods etc. It's what you were able to devise from experiments, and what I should be able to observe when reproducing your experiment.

This thread has devolved quickly into the Frequentist vs Bayesian debate, and that is not easily resolvable. The math in both approaches is solid, so it always comes down to philosophical preferences. The frequentist interpretation of probability as the limit of an event's relative frequency is justified by the strong law of large numbers; regardless of your preferred interpretation of probability, an event's relative frequency will converge to its probability with probability 1.

Frequentist confidence intervals are indeed trickier to interpret than Bayesian credible intervals. By treating an unknown quantity as a random variable, Bayesians can assert that one interval contains that quantity with some probability. Frequentists refuse to treat some quantities as random variables, and any equations containing only constants can only be true or false. So when estimating an unknown constant, frequentists must bound them with a RANDOM interval to involve probability at all. Rather than one interval containing a random variable with some probability, a frequentist method generates many different possible intervals, some of which contains the unknown constant. If the coverage probability is reasonably high, it's a reasonable leap of faith to assert that a particular interval contains the unknown constant (note, not "with some probability").

A Bayesian would balk at such a leap of faith as much as a Frequentist balks at treating any unknown quantity as a random variable. The frequentist Neyman construction method in fact exposed an embarrassing issue with such leaps of faith. Without actively preventing it (see Feldman and Cousins, 1997 for one approach), rare outcomes may generate EMPTY confidence intervals for a distribution parameter. Such a leap of faith would be very unreasonable! I've seen a few Bayesians using that example to mock frequentist methods, while frequentists typically respond with "well I still get a correct interval most of the time, and without making false assumptions." I'll point out that the Bayesian/frequentist impasse is not important to most who apply their methods. Even people who are committed to coverage probability will often use Bayesian methods if the methods are shown to have good coverage probability in simulations.