# Are all values within a 95% confidence interval equally likely?

I have found discordant information on the question: "If one constructs a 95% confidence interval (CI) of a difference in means or a difference in proportions, are all values within the CI equally likely? Or, is the point estimate the most likely, with values near the "tails" of the CI less likely than those in the middle of the CI?

For instance, if a randomized clinical trial report states that the relative risk of mortality with a particular treatment is 1.06 (95% CI 0.96 to 1.18), is the likelihood of 0.96 being the correct value the same as 1.06?

I found many references to this concept online, but the following two examples reflect the uncertainty therein:

1. Lisa Sullivan's module about Confidence Intervals states:

The confidence intervals for the difference in means provide a range of likely values for ($μ_1-μ_2$). It is important to note that all values in the confidence interval are equally likely estimates of the true value of ($μ_1-μ_2$).

2. This blogpost, titled Within the Margin of Error, states:

What I have in mind is misunderstanding about “margin of error” that treats all points within the confidence interval as equally likely, as if the central limit theorem implied a bounded uniform distribution instead of a t distribution. [...]
The thing that talk about “margin of error” misses is that possibilities that are close to the point estimate are much more likely than possibilities that are at the edge of the margin".

These seem contradictory, so which is correct?

-
I wonder if there is confusion somewhere with the related concept that p-values are uniformly distributed under the null hypothesis.. –  Michael McGowan Oct 19 '12 at 19:01
The first quotation is an aberrant slip in an otherwise accurate account of confidence intervals. The second quotation is from an account that, to put it nicely, is a sloppy mess: it's full of statements that are vague, incorrect, or can only be interpreted in a Bayesian sense. But both quotations are wrong! –  whuber Oct 19 '12 at 19:18
@whuber I wouldn't call the second one a mess...I'd call it a Bayesian interpretation of the Frequentist interpretation :) –  Michael McGowan Oct 19 '12 at 19:25
@Michael One example of sloppiness is a solecism like asserting the CLT implies an "infinite number of repeated estimates of [the population] mean will still follow a normal distribution." One does not have to be wrong in order to communicate ideas simply to a nontechnical audience. –  whuber Oct 19 '12 at 19:30
@whuber, I consider the sentence you cite only a minor sin. The main error is that CLT does not involve t distribution. –  glassy Jan 15 '13 at 8:39

One question that needs to be answered is what does "likely" mean in this context?

If it means probability (as it is sometimes used as a synonym of) and we are using strict frequentist definitions then the true parameter value is a single value that does not change, so the probability (likelihood) of that point is 100% and all other values are 0%. So almost all are equally likely at 0%, but if the interval contains the true value, then it is different from the others.

If we use a Bayesian approach then the CI (Credible Interval) comes from the posterior distribution and you can compare the likelihood at the different points within the interval. Unless the posterior is perfectly uniform within the interval (theoretically possible I guess, but that would be a strange circumstance) then the values have different likelihoods.

If we use likely to be similar to confidence then think about it this way: Compute a 95% confidence interval, a 90% confidence interval, and an 85% confidence interval. We would be 5% confident that the true value lies in the region inside of the 95% interval but outside of the 90% interval, we could say that the true value is 5% likely to fall in that region. The same is true for the region that is inside the 90% interval but outside the 85% interval. So if every value is equally likely, then the size of the above 2 regions would need to be exactly the same and the same would hold true for the region inside a 10% confidence interval but outside a 5% confidence interval. None of the standard distributions that intervals are constructed using have this property (except special cases with 1 draw from a uniform).

You could further prove this to yourself by simulating a large number of datasets from known populations, computing the confidence interval of interest, then comparing how often the true parameter is closer to the point estimate than to each of the end points.

-
Likelihood is what this question needs in the answer, not probability, either frequentist or Bayesian. Likelihood provides exactly the answer, the others can only do so with some twisting and stretching. –  Michael Lew Oct 20 '12 at 7:21
@Greg I like your explanation. Just to be clear, your argument supports the notion that the values at the "tails" of the 95% CI are less likely (less probable) than those closer to the point estimate, correct? Thanks for your response. –  pmgjones Oct 21 '12 at 14:15
@pmgjones less probable, NO, see 2nd paragraph. Less likely in the context of the 4th paragraph, Yes. –  Greg Snow Oct 22 '12 at 17:35
@GregSnow Your 2nd paragraph says almost exactly that the probability of the true parameter being the true parameter is 100%. Do you really believe that this tautology is what "strict frequentist definitions" have to offer? –  rolando2 Jan 26 '13 at 4:13
@rolando2, I think that frequentist statistics has much to offer, I was just eliminating the common misstatements that imply the true value changes and is sometimes out of the interval and sometimes inside the interval (and sometimes closer to the boundaries and sometimes closer to the center). The later paragraphs then get at the more true feel for the ideas. –  Greg Snow Jan 26 '13 at 16:11

This is a great question! There is a mathematical concept called likelihood that will help you understand the issues. Fisher invented likelihood but considered it to be somewhat less desirable than probability, but likelihood turns out to be more 'primitive' than probability and Ian Hacking (1965) considered it to be axiomatic in that it is not provable. Likelihood underpins probability rather then the reverse.

Hacking, 1965. Logic of Statistical Inference.

Likelihood is not given the attention that it should have in standard textbooks of statistics, for no good reason. It differs from probability in having almost exactly the properties that one would expect, and likelihood functions and intervals are very useful for inference. Perhaps likelihood is not liked by some statisticians because there is sometimes no 'proper' way to derive the relevant likelihood functions. However, in many cases the likelihood functions are obvious and well defined. A study of likelihoods for inference should probably start with Richard Royall's small and easy to understand book called Statistical Evidence: a Likelihood Paradigm.

The answer to your question is that no, the points within any interval do not all have the same likelihood. Those at the edges of a confidence interval usually have lower likelihoods than others towards the centre of the interval. Of course, the conventional confidence interval tells you nothing directly about the parameter relevant to the particular experiment. Neyman's confidence intervals are 'global' in that they are designed to have long-run properties rather than 'local' properties relevant to the experiment in hand. (Happily good long-run performance can be interpreted in the local, but that is an intellectual shortcut rather than a mathematical reality.) Likelihood intervals—in the cases where they can be constructed—directly reflect the likelihood relating the experiment in hand. There is less about likelihood intervals that is confusing than is the case for confidence intervals, in my opinion, and they have more utility than might be expected from their almost complete disuse!

-
@suncoolsu It is not necessary that the interval in question be a likelihood interval for the statement to be true. The interval only has to span the most likely estimate so that the interval bounds are each less likely than a point within the interval. Any ordinary confidence interval will satisfy that requirement. –  Michael Lew Oct 21 '12 at 6:48
@pmjones A 95% CI DOEST NOT tell you if the values towards the margins of the CI are closer to the truth than the values in the middle. CIs make statements about repeated sampling from the population. In the long run (i.e., after repeated sampling), 95% of such CIs, which are constructed for each sample, will cover the true value. Therefore, there are two key observations 1) One cannot say anything about the true value for a given CI 2) CIs do not tell you anything about the observed data, which is a usual Bayesian criticism. –  suncoolsu Oct 21 '12 at 19:09
@MichaelLew Likelihood principle is useful, but I was saying that (quoting LW) "Indeed, all of frequentist inference violates LP, so if we adhered to LP we would have to abandon frequentist inference." Because CI is a frequentist idea, it violates LP (which you say is fundamental). –  suncoolsu Oct 21 '12 at 19:12
@suncollsu The question is not whether a confidence interval alone and without any other statistical considerations tells anything about the likelihood of parameter values within itself. It is about the likelihood of parameter values within the interval. The likelihood function answers the question, and that answer is correct even if the confidence interval violates the likelihood principle. (Read my previous comment again. You seem to have ignored its content entirely.) –  Michael Lew Oct 21 '12 at 20:13
@rolando2 Neyman's 95% confidence intervals are designed so that the method contains the true parameter on 95% of occasions that the method is used. Strictly speaking the confidence attaches to the method and not to any individual interval and so the individual interval does not tell you anything about the state of the world in that particular experiment. See my answer to this question for more detail: stats.stackexchange.com/questions/8844/… –  Michael Lew Jan 27 '13 at 20:05

Suppose someone told me that I should place equal trust in all values within a CI95 as potential indicators of the population value. (I'm deliberately avoiding the terms "likely" and "probable.") What's special about 95? Nothing: to be consistent I would also have to place equal trust in all values within a CI96, a CI97, ... and a CI99.9999999. As the CI's coverage approached its limit, virtually all real numbers would have to be included. The preposterousness of this conclusion would lead me to reject the initial claim.

-
This is a great answer! I should have thought of the effect of approaching extremes of possible CIs. Thanks for writing this! –  pmgjones Jan 26 '13 at 13:45

Let's start with the definition of a confidence interval. If I say that a 95% confidence interval goes from this to that I mean that statements of that nature will be true about 95% of the time and false about 5% of the time. I do not necessarily mean that I am 95% confident about this particular statement. A 90% confidence interval will be narrower and an 80% narrower still. Therefore, when wondering what the true value is, I have less credence in values as they get closer and closer to the edge of any particular confidence interval.

Note that all of the above is qualitative, especially "credence". (I avoided the term "confidence" or "likelihood" in that statement because they carry mathematical baggage that may differ from our intuitive baggage.) Bayesian approaches would rephrase your question to something that has a quantitative answer but I don't want to open that can of worms here.

Box, Hunter & Hunter's classic text ("Statistics for Experimenters", Wiley, 1978) may also help. See "Sets of Confidence Intervals" on pp 113, ff.

-
Since we are dealing partly in concepts and partly in semantics, I'll point out that in your second sentence you've said "...statements of that nature will be true..." without specifying what statements would be true. –  rolando2 Jan 26 '13 at 23:40

If likely means the probability the parameter is X, then yes, all values in the confidence interval are equally likely. Furthermore, if your confidence interval is say, (2, 4), the Frequentist approach cannot say which is more likely to be the true parameter, 3.4 or 4.2, even though 4.2 is not even in the interval.

It was issues such as these that Fisher sought to try something new, mainly Fidicual inference. While some call this his biggest blunder, Brad Efron has commented, correctly I believe, it (or something like it) may be the 21st century's biggest hit. There is renewed interest in new approaches to Statistical Inference. Inferential Models and Confidence Distributions are two that come to mind.

Simulation: Frequentist confidence intervals are long run probability interpretations as mentioned by @suncoolsu. We cannot conclude anything from the single confidence interval on hand. To demonstrate I ran a simulation of the simplest problem I could think of, estimating a 95% CI for the mean, $\mu$, from one observation of $N(\mu,1)$.

Set $\mu=0$:

set.seed(1)
x=rnorm(1)
print(x)
[1] -0.626454
print(c(x-2,x+2))
[1] -2.626454  1.373546


Can we say now the probability $\mu= 1.3$ is larger than $\mu= 1.4$? Or probability $\mu= -.62$ is larger than $\mu = 0$? If so what are these probabilities? We simply cannot say anything about which values in a confidence interval from one sample are more likely to be the true parameter $\mu$ (or whether values in the interval are more likely to be that parameter than values outside the interval).

Edit: I understand my answer is not a popular one, and there are good arguments being made to counter my point. I have not deleted my answer so I can at least raise an objection to the popular answer.

I will slowly provide references that support my point of view (including an upcoming JASA paper that has been accepted but not published yet). I hope eventually my response will at least be non-negative.

1) I agree with the CI section of this commentary that supports my view: http://wmbriggs.com/blog/?p=3169

-
@Glen - Michael Lew's answer applies to any type of interval that has bounds that are either side of the parameter value that has the highest likelihood. In such a case, the fact that the intervals are confidence intervals rather than likelihood intervals doesn't matter at all and the centre of most confidence intervals are probably quite close the the most likely parameter value. The question is properly couched in terms of likelihood, and the answer can only be directly given in terms of likelihoods. –  Michael Lew Oct 20 '12 at 7:26
Your statement is wrong (-1, sorry). Likelihood may be unfamiliar to you, but it is the statistical principle that is appropriate for this question. –  Michael Lew Oct 20 '12 at 7:28
No the answer is not wrong. I will supply references soon. –  Glen Oct 20 '12 at 11:25
I agree with @Michael: all the quantitative ways I can think of interpreting "likely" contradict the first sentence of this answer. –  whuber Oct 20 '12 at 16:49
I think Glen is right when he says all values in a Confidence Interval are equally likely. CIs don't tell which value in the CI is closer to the "truth". Under repeated sampling they will cover the true value with certain probability, but no judgment could be made about the data in hand, or location of "truth" in CI. –  suncoolsu Oct 21 '12 at 2:22