When are confidence intervals useful?

Question

If I understand correctly a confidence interval of a parameter is an interval constructed by a method which yields intervals containing the true value for a specified proportion of samples. So the 'confidence' is about the method rather than the interval I compute from a particular sample.

As a user of statistics I have always felt cheated by this since the space of all samples is hypothetical. All I have is one sample and I want to know what that sample tells me about a parameter.

Is this judgement wrong? Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?

[This question arises from second thoughts after dissing confidence intervals in a math.se answer https://math.stackexchange.com/questions/7564/calculating-a-sample-size-based-on-a-confidence-level/7572#7572 ]

Answer 1

I like to think of CIs as some way to escape the Hypothesis Testing (HT) framework, at least the binary decision framework following Neyman's approach, and keep in line with theory of measurement in some way. More precisely, I view them as more close to the reliability of an estimation (a difference of means, for instance), and conversely HT are more close to hypothetico-deductive reasoning, with its pitfalls (we cannot accept the null, the alternative is often stochastic, etc.). Still, with both interval estimation and HT we have to rely on distribution assumptions most of the time (e.g. a sampling distribution under $H_0$), which allows to make inference from our sample to the general population or a representative one (at least in the frequentist approach).

In many context, CIs are complementary to usual HT, and I view them as in the following picture (it is under $H_0$):

that is, under the HT framework (left), you look at how far your statistic is from the null, while with CIs (right) you are looking at the null effect "from your statistic", in a certain sense.

Also, note that for certain kind of statistic, like odds-ratio, HT are often meaningless and it is better to look at its associated CI which is assymmetrical and provide more relevant information as to the direction and precision of the association, if any.

Answer 2

An alternative approach relevant to your 2nd Q, "Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?":

You should take a look at Bayesian inference and the resulting credible intervals. A 95% credible interval can be interpreted as an interval which you believe has 95% probability of including the true parameter value. The price you pay is that you need to put a prior probability distribution on the values you believe the true parameter is likely to take before collecting the data. And your prior may differ from someone else's prior, so your resulting credible intervals may also differ even when you use the same data.

This is only my quick and crude attempt to summarise! A good recent textbook with a practical focus is:

Andrew Gelman, John B. Carlin, Hal S. Stern and Donald B. Rubin. "Bayesian Data Analysis" (2nd edition). Chapman & Hall/CRC, 2003. ISBN 978-1584883883

Answer 3

You are correct in saying that the 95% confidence intervals are things that result from using a method that works in 95% of cases, rather than any individual interval having a 95% likelihood of containing the expected value.

"The logical basis and interpretation of confidence limits are, even now, a matter of controversy." {David Colquhoun, 1971, Lectures on Biostatistics}

That quotation is taken from a statistics textbook published in 1971, but I would contend that it is still true in 2010. The controversy is probably most extreme in the case of confidence intervals for binomial proportions. There are many competing methods for calculating those confidence intervals, but they are all inaccurate in one or more senses and even the worst performing method has proponents among textbook authors. Even so called ‘exact’ intervals fail to yield the properties expected of confidence intervals.

In a paper written for surgeons (widely known for their interest in statistics!), John Ludbrook and I argued for the routine use of confidence intervals calculated using a uniform Bayesian prior because such intervals have frequentist properties as good as any other method (on average exactly 95% coverage over all true proportions) but, importantly, much better coverage over all observed proportions (exactly 95% coverage). The paper, because of its target audience, is not terribly detailed and so it may not convince all statistician, but I am working on a follow-up paper with the full set of results and justifications.

This is a case where the Bayesian approach has frequentist properties as good as the frequentist approach, something that happens fairly often. The assumption of a uniform prior is not problematical because a uniform distribution of population proportions is built into every calculation of frequentist coverage that I've come across.

You ask: "Are there ways of looking at confidence intervals, at least in some circumstances, which would be meaningful to users of statistics?" My answer, then, is that for binomial confidence intervals one can get intervals that contain the population proportion exactly 95% of the time for all observed proportions. That is a yes. However, the conventional use of confidence intervals expects coverage for all population proportions and for that the answer is "No!"

The length of the answers to your question, and the various responses to them suggests that confidence intervals are widely misunderstood. If we change our objective from coverage for all true parameter values to coverage of the true parameter value for all sample values, it might get easier because the intervals will then be shaped to be directly relevant to the observed values rather than for the performance of the method per se.

Answer 4

I think the premise of this question is flawed because it denies the distinction between the uncertain and the known.

Describing a coin flip provides a good analogy. Before the coin is flipped, the outcome is uncertain; afterwards, it is no longer "hypothetical." Confusing this fait accompli with the actual situation we wish to understand (the behavior of the coin, or decisions that are to be made as a result of its outcome) essentially denies a role for probability in understanding the world.

This contrast is thrown in sharp relief within an experimental or regulatory arena. In such cases the scientist or the regulator know they will be faced with situations whose outcomes, at any time beforehand, are unknown, yet they must make important determinations such as how to design the experiment or establish the criteria to use in determining compliance with regulations (for drug testing, workplace safety, environmental standards, and so on). These people and the institutions for which they work need methods and knowledge of the probabilistic characteristics of those methods in order to develop optimal and defensible strategies, such as good experimental designs and fair decision procedures that err as little as possible.

Confidence intervals, despite their classically poor justification, fit into this decision-theoretic framework. When a method of constructing a random interval has a combination of good properties, such as assuring a minimal expected coverage of the interval and minimizing the expected length of the interval--both of them a priori properties, not a posteriori ones--then over a long career of using that method we can minimize the costs associated with the actions that are indicated by that method.

Answer 5

This is a great discussion. I feel that Bayesian credible intervals and likelihood support intervals are the way to go, as well as Bayesian posterior probabilities of events of interest (e.g., a drug is efficacious). But supplanting P-values with confidence intervals is a major gain. Virtually every issue of the finest medical journals such as NEJM and JAMA has a paper with the "absence of evidence is not evidence of absence" problem in their abstracts. The use of confidence intervals will largely prevent such blunders. A great little text is http://www.amazon.com/Statistics-Confidence-Intervals-Statistical-Guidelines/dp/0727913751

Answer 6

To address your question directly: Suppose that you are contemplating the use of a machine to fill a cereal box with a certain amount of cereal. Obviously, you do not want to overfill/underfill the box. You want to assess the reliability of the machine. You perform a series of tests like so: (a) Use the machine to fill the box and (b) Measure the amount of cereal that is filled in the box.

Using the data collected you construct a confidence interval for the amount of cereal that the machine is likely to fill in the box. This confidence interval tells us that the interval we obtained has a 95% probability that it will contain the the true amount of cereal the machine will put in the box. As you say, the interpretation of the confidence interval relies on hypothetical, unseen samples generated by the method under consideration. But, this is precisely what we want in our context. In the above context, we will use the machine repeatedly to fill the box and thus we care about hypothetical, unseen realizations of the amount of cereal the machine fills in the box.

To abstract away from the above context: a confidence interval gives us a guarantee that if we were to use the method under investigation (in the above example method = machine) repeatedly there is a 95% probability that the confidence interval will have the true parameter.