# When are confidence intervals useful?

If I understand correctly a confidence interval of a parameter is an interval constructed by a method which yields intervals containing the true value for a specified proportion of samples. So the 'confidence' is about the method rather than the interval I compute from a particular sample.

As a user of statistics I have always felt cheated by this since the space of all samples is hypothetical. All I have is one sample and I want to know what that sample tells me about a parameter.

Is this judgement wrong? Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?

[This question arises from second thoughts after dissing confidence intervals in a math.se answer https://math.stackexchange.com/questions/7564/calculating-a-sample-size-based-on-a-confidence-level/7572#7572 ]

I like to think of CIs as some way to escape the Hypothesis Testing (HT) framework, at least the binary decision framework following Neyman's approach, and keep in line with theory of measurement in some way. More precisely, I view them as more close to the reliability of an estimation (a difference of means, for instance), and conversely HT are more close to hypothetico-deductive reasoning, with its pitfalls (we cannot accept the null, the alternative is often stochastic, etc.). Still, with both interval estimation and HT we have to rely on distribution assumptions most of the time (e.g. a sampling distribution under $H_0$), which allows to make inference from our sample to the general population or a representative one (at least in the frequentist approach).

In many context, CIs are complementary to usual HT, and I view them as in the following picture (it is under $H_0$):

that is, under the HT framework (left), you look at how far your statistic is from the null, while with CIs (right) you are looking at the null effect "from your statistic", in a certain sense.

Also, note that for certain kind of statistic, like odds-ratio, HT are often meaningless and it is better to look at its associated CI which is assymmetrical and provide more relevant information as to the direction and precision of the association, if any.

• Why do you say hypothesis tests are often meaningless for odds ratios, any more than any other effect estimate? I'd stress instead that confidence intervals are more useful than standard errors for odds ratios and other estimates with asymmetric sampling distributions in finite samples. – onestop Oct 23 '10 at 8:53
• @onestop Well, I was partly thinking of what you say about "asymmetric sampling distributions..." (and it seems I was not so clear), but also of the fact that in epidemiological studies we are generally most interested in CIs (that is, how precise is our estimate) than HT. – chl Oct 23 '10 at 12:33
• +1. This reminds me that I've been using your scripts to learn asymptote by jumping in and changing stuff around, trying different things. Thanks again for that, very helpful to get started. – ars Oct 30 '10 at 3:07
• @ars Actually, I seem to remember that this picture was made with PStricks. Anyway, a good starting point for Asymptote is piprime.fr/asymptote. – chl Oct 30 '10 at 8:59
• @chl, this may be off-topic, but can you please tell me if you made these graphs in R? – suncoolsu Nov 3 '10 at 1:41

An alternative approach relevant to your 2nd Q, "Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?":

You should take a look at Bayesian inference and the resulting credible intervals. A 95% credible interval can be interpreted as an interval which you believe has 95% probability of including the true parameter value. The price you pay is that you need to put a prior probability distribution on the values you believe the true parameter is likely to take before collecting the data. And your prior may differ from someone else's prior, so your resulting credible intervals may also differ even when you use the same data.

This is only my quick and crude attempt to summarise! A good recent textbook with a practical focus is:

Andrew Gelman, John B. Carlin, Hal S. Stern and Donald B. Rubin. "Bayesian Data Analysis" (2nd edition). Chapman & Hall/CRC, 2003. ISBN 978-1584883883

• Thanks. But what about frequentist confidence intervals specifically? Are there any circumstances at all where they would be relevant? – Jyotirmoy Bhattacharya Oct 23 '10 at 9:33
• I believe having different priors is a non issue (at least from the objective Bayesian point of view), if it happens that you have different knowledge about the situation at hand. We meed to see the priors as a way of casting our a priori information. I know that it is not simple... – teucer Oct 23 '10 at 14:57
• @Jyotirmoy About bayesian vs. frequentist approaches, interesting points were made here: stats.stackexchange.com/questions/1611/… – chl Oct 23 '10 at 15:23

I think the premise of this question is flawed because it denies the distinction between the uncertain and the known.

Describing a coin flip provides a good analogy. Before the coin is flipped, the outcome is uncertain; afterwards, it is no longer "hypothetical." Confusing this fait accompli with the actual situation we wish to understand (the behavior of the coin, or decisions that are to be made as a result of its outcome) essentially denies a role for probability in understanding the world.

This contrast is thrown in sharp relief within an experimental or regulatory arena. In such cases the scientist or the regulator know they will be faced with situations whose outcomes, at any time beforehand, are unknown, yet they must make important determinations such as how to design the experiment or establish the criteria to use in determining compliance with regulations (for drug testing, workplace safety, environmental standards, and so on). These people and the institutions for which they work need methods and knowledge of the probabilistic characteristics of those methods in order to develop optimal and defensible strategies, such as good experimental designs and fair decision procedures that err as little as possible.

Confidence intervals, despite their classically poor justification, fit into this decision-theoretic framework. When a method of constructing a random interval has a combination of good properties, such as assuring a minimal expected coverage of the interval and minimizing the expected length of the interval--both of them a priori properties, not a posteriori ones--then over a long career of using that method we can minimize the costs associated with the actions that are indicated by that method.

• Give a an example of using a confidence interval to make a decision. Or, better yet, compare two confidence intervals and how you would make different decisions with each one, while keeping completely with in the frequentist framework. – BrainPermafrost Feb 24 '19 at 1:10
• @Brain Any introductory stats textbook will provide such examples. One that is unabashedly frequentist is Freedman, Pisani, and Purves, Statistics (any edition). – whuber Feb 24 '19 at 16:15

You are correct in saying that the 95% confidence intervals are things that result from using a method that works in 95% of cases, rather than any individual interval having a 95% likelihood of containing the expected value.

"The logical basis and interpretation of confidence limits are, even now, a matter of controversy." {David Colquhoun, 1971, Lectures on Biostatistics}

That quotation is taken from a statistics textbook published in 1971, but I would contend that it is still true in 2010. The controversy is probably most extreme in the case of confidence intervals for binomial proportions. There are many competing methods for calculating those confidence intervals, but they are all inaccurate in one or more senses and even the worst performing method has proponents among textbook authors. Even so called ‘exact’ intervals fail to yield the properties expected of confidence intervals.

In a paper written for surgeons (widely known for their interest in statistics!), John Ludbrook and I argued for the routine use of confidence intervals calculated using a uniform Bayesian prior because such intervals have frequentist properties as good as any other method (on average exactly 95% coverage over all true proportions) but, importantly, much better coverage over all observed proportions (exactly 95% coverage). The paper, because of its target audience, is not terribly detailed and so it may not convince all statistician, but I am working on a follow-up paper with the full set of results and justifications.

This is a case where the Bayesian approach has frequentist properties as good as the frequentist approach, something that happens fairly often. The assumption of a uniform prior is not problematical because a uniform distribution of population proportions is built into every calculation of frequentist coverage that I've come across.

You ask: "Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?" My answer, then, is that for binomial confidence intervals one can get intervals that contain the population proportion exactly 95% of the time for all observed proportions. That is a yes. However, the conventional use of confidence intervals expects coverage for all population proportions and for that the answer is "No!"

The length of the answers to your question, and the various responses to them suggests that confidence intervals are widely misunderstood. If we change our objective from coverage for all true parameter values to coverage of the true parameter value for all sample values, it might get easier because the intervals will then be shaped to be directly relevant to the observed values rather than for the performance of the method per se.

This is a great discussion. I feel that Bayesian credible intervals and likelihood support intervals are the way to go, as well as Bayesian posterior probabilities of events of interest (e.g., a drug is efficacious). But supplanting P-values with confidence intervals is a major gain. Virtually every issue of the finest medical journals such as NEJM and JAMA has a paper with the "absence of evidence is not evidence of absence" problem in their abstracts. The use of confidence intervals will largely prevent such blunders. A great little text is http://www.amazon.com/Statistics-Confidence-Intervals-Statistical-Guidelines/dp/0727913751

To address your question directly: Suppose that you are contemplating the use of a machine to fill a cereal box with a certain amount of cereal. Obviously, you do not want to overfill/underfill the box. You want to assess the reliability of the machine. You perform a series of tests like so: (a) Use the machine to fill the box and (b) Measure the amount of cereal that is filled in the box.

Using the data collected you construct a confidence interval for the amount of cereal that the machine is likely to fill in the box. This confidence interval tells us that the interval we obtained has a 95% probability that it will contain the the true amount of cereal the machine will put in the box. As you say, the interpretation of the confidence interval relies on hypothetical, unseen samples generated by the method under consideration. But, this is precisely what we want in our context. In the above context, we will use the machine repeatedly to fill the box and thus we care about hypothetical, unseen realizations of the amount of cereal the machine fills in the box.

To abstract away from the above context: a confidence interval gives us a guarantee that if we were to use the method under investigation (in the above example method = machine) repeatedly there is a 95% probability that the confidence interval will have the true parameter.

• @Srikant. No! This is how classical CIs bite. Let's assume for simplicity that the amount of cereal filled in a box is normal with mean $\mu$ and variance $\sigma^2$. The confidence interval of $\mu$ is based on its sampling distribution which is different. A particular CI may be way off due to sampling errors and then it will have no relation to how the machine performs. If you were to repeatedly sample and repeatedly form CIs then 95% of them would be right, but that is no consolation. – Jyotirmoy Bhattacharya Oct 23 '10 at 13:44
• @Jyotirmoy Of course, a specific CI may be way-off. In other words, there is a 5% chance that the CI does not contain the true value. Nevertheless, the interpretation I gave is consistent with how CIs are actually constructed. We imagine using the method repeatedly and construct the CI such that the probability that the observed CI contains the true value is 0.95. Notice that my answer does not say anything about the probability of where the true value actually lies as that is a statement that can only be made with credible intervals and not confidence intervals. – user28 Oct 23 '10 at 14:03
• @Jyotirmoy Lower/Upper bounds for a $(100-\alpha)$% CI of an observed mean are constructed under $H_0$, where the sampling distribution of a mean (or a difference of means) is the one you assumed depending on your sample ($t$ or $z$ distribution). I found Srikant's answer correct, and his interpretation doesn't seem to go beyond the experiment that was framed. CIs are random variables. – chl Oct 23 '10 at 14:09
• @Srikant. I perhaps misunderstood "method=machine" in the answer. I thought you were saying that 95% of all boxes coming out of the assembly line would have weights within the 95% confidence interval derived from a particular sample of the boxes. – Jyotirmoy Bhattacharya Oct 23 '10 at 14:43