Are all values within a 95% confidence interval equally likely?

Question

I have found discordant information on the question: "If one constructs a 95% confidence interval (CI) of a difference in means or a difference in proportions, are all values within the CI equally likely? Or, is the point estimate the most likely, with values near the "tails" of the CI less likely than those in the middle of the CI?

For instance, if a randomized clinical trial report states that the relative risk of mortality with a particular treatment is 1.06 (95% CI 0.96 to 1.18), is the likelihood of 0.96 being the correct value the same as 1.06?

I found many references to this concept online, but the following two examples reflect the uncertainty therein:

1. Lisa Sullivan's module about Confidence Intervals states:

   The confidence intervals for the difference in means provide a range of likely values for ($μ_1-μ_2$). It is important to note that all values in the confidence interval are equally likely estimates of the true value of ($μ_1-μ_2$).

2. This blogpost, titled Within the Margin of Error, states:

   What I have in mind is misunderstanding about “margin of error” that treats all points within the confidence interval as equally likely, as if the central limit theorem implied a bounded uniform distribution instead of a t distribution. [...]
   The thing that talk about “margin of error” misses is that possibilities that are close to the point estimate are much more likely than possibilities that are at the edge of the margin".

These seem contradictory, so which is correct?

Answer 1

One question that needs to be answered is what does "likely" mean in this context?

If it means probability (as it is sometimes used as a synonym of) and we are using strict frequentist definitions then the true parameter value is a single value that does not change, so the probability (likelihood) of that point is 100% and all other values are 0%. So almost all are equally likely at 0%, but if the interval contains the true value, then it is different from the others.

If we use a Bayesian approach then the CI (Credible Interval) comes from the posterior distribution and you can compare the likelihood at the different points within the interval. Unless the posterior is perfectly uniform within the interval (theoretically possible I guess, but that would be a strange circumstance) then the values have different likelihoods.

If we use likely to be similar to confidence then think about it this way: Compute a 95% confidence interval, a 90% confidence interval, and an 85% confidence interval. We would be 5% confident that the true value lies in the region inside of the 95% interval but outside of the 90% interval, we could say that the true value is 5% likely to fall in that region. The same is true for the region that is inside the 90% interval but outside the 85% interval. So if every value is equally likely, then the size of the above 2 regions would need to be exactly the same and the same would hold true for the region inside a 10% confidence interval but outside a 5% confidence interval. None of the standard distributions that intervals are constructed using have this property (except special cases with 1 draw from a uniform).

You could further prove this to yourself by simulating a large number of datasets from known populations, computing the confidence interval of interest, then comparing how often the true parameter is closer to the point estimate than to each of the end points.

Answer 2

Suppose someone told me that I should place equal trust in all values within a CI95 as potential indicators of the population value. (I'm deliberately avoiding the terms "likely" and "probable.") What's special about 95? Nothing: to be consistent I would also have to place equal trust in all values within a CI96, a CI97, ... and a CI99.9999999. As the CI's coverage approached its limit, virtually all real numbers would have to be included. The preposterousness of this conclusion would lead me to reject the initial claim.

Answer 3

This is a great question! There is a mathematical concept called likelihood that will help you understand the issues. Fisher invented likelihood but considered it to be somewhat less desirable than probability, but likelihood turns out to be more 'primitive' than probability and Ian Hacking (1965) considered it to be axiomatic in that it is not provable. Likelihood underpins probability rather then the reverse.

Hacking, 1965. Logic of Statistical Inference.

Likelihood is not given the attention that it should have in standard textbooks of statistics, for no good reason. It differs from probability in having almost exactly the properties that one would expect, and likelihood functions and intervals are very useful for inference. Perhaps likelihood is not liked by some statisticians because there is sometimes no 'proper' way to derive the relevant likelihood functions. However, in many cases the likelihood functions are obvious and well defined. A study of likelihoods for inference should probably start with Richard Royall's small and easy to understand book called Statistical Evidence: a Likelihood Paradigm.

The answer to your question is that no, the points within any interval do not all have the same likelihood. Those at the edges of a confidence interval usually have lower likelihoods than others towards the centre of the interval. Of course, the conventional confidence interval tells you nothing directly about the parameter relevant to the particular experiment. Neyman's confidence intervals are 'global' in that they are designed to have long-run properties rather than 'local' properties relevant to the experiment in hand. (Happily good long-run performance can be interpreted in the local, but that is an intellectual shortcut rather than a mathematical reality.) Likelihood intervals—in the cases where they can be constructed—directly reflect the likelihood relating the experiment in hand. There is less about likelihood intervals that is confusing than is the case for confidence intervals, in my opinion, and they have more utility than might be expected from their almost complete disuse!

Answer 4

Although the phrase “all values within the CI equally likely” is unclear in its meaning, it would seem to imply that our confidences in equal-length segments of the CI should be the same irrespective of where in the CI they are located. However this implication is false. To demonstrate this, consider the example with two normally distributed random variables with known variances, and CI based on:

In summary, for this example, confidence per unit length near the outer boundaries of the CI is much less than it is near the centre of the CI.

Answer 5

Let's start with the definition of a confidence interval. If I say that a 95% confidence interval goes from this to that I mean that statements of that nature will be true about 95% of the time and false about 5% of the time. I do not necessarily mean that I am 95% confident about this particular statement. A 90% confidence interval will be narrower and an 80% narrower still. Therefore, when wondering what the true value is, I have less credence in values as they get closer and closer to the edge of any particular confidence interval.

Note that all of the above is qualitative, especially "credence". (I avoided the term "confidence" or "likelihood" in that statement because they carry mathematical baggage that may differ from our intuitive baggage.) Bayesian approaches would rephrase your question to something that has a quantitative answer but I don't want to open that can of worms here.

Box, Hunter & Hunter's classic text ("Statistics for Experimenters", Wiley, 1978) may also help. See "Sets of Confidence Intervals" on pp 113, ff.

Answer 6

1. I recommend reading the other answers, I thought the answer by Greg Snow was excellent, and the discussion by Michael Lew on likelihood very interesting.
2. I want to encourage people to make use of tools like R and Python to explore (counter) examples and develop their intuition. This is what we will do in this post, essentially the simulation suggested at the end of Greg's answer.

The plan

In Python, we will create a normal distribution, where the true mean is $\mu=0$ and the standard deviation is $\sigma=1$.

• We will repeatedly take random samples from this distribution and calculate 95% confidence intervals for the mean. To do this we will use the t-distribution (this method is commonly taught).
• These intervals are symmetric, and the midpoint is the sample mean $\bar{x}$.
• The confidence intervals we generate will have different widths, depending on the sample (in particular the sample standard deviation).
• For any given sample say the width of the confidence interval is $2w$, so the distance from the midpoint(the sample mean $\bar{x}$) to the end of the interval is $w$.
• We will compute the distance from the interval midpoint to the true mean, and then divide by the width $w$, to put this distance in terms of $w$. Call this $d$. That is $d = \frac{(\bar{x} - \mu)}{w}$. We do this normalisation because the confidence intervals could have different widths.
• Finally we will plot the distribution of these $d$s.
• We will show that if we repeat the simulation many times, that the true mean is more frequently closer to the midpoints of the confidence intervals than it is to either of the endpoints.

Simulation Plots

We plot first a histogram of the distances $d$, and then the cumulative distribution of the (absolute) distance $|d|$

Highlights

• We see from the simulation that values of $d$ around 0 occur most frequently (that is, the true mean is close to the midpoint), and larger distances $d$ occur more rarely.
• When the absolute distance is less than $\frac{w}{2}$, it means the true mean is closer to the interval midpoint than either endpoint. In the cumulative plot, we can see that happens about 70% of the time.
• Side note: The bars shaded red in the histogram are when the distance is greater than $w$, this is when the true mean is not covered by the interval. In this case it happens about 5% of the time (recall we were generating 95% confidence intervals).

Code

The core part of the Python code (not including the plotting) is copied below.

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

# Parameters
n_simulations = 10000
sample_size = 30
true_mean = 0
std_dev = 1

# Store the distance from the true mean in terms of w for each simulation
distance_in_terms_of_w = []

# Run simulations
for _ in range(n_simulations):
    # Generate sample
    sample = np.random.normal(loc=true_mean, scale=std_dev, size=sample_size)
    
    # Calculate sample mean and standard error
    sample_mean = np.mean(sample)
    standard_error = stats.sem(sample)
    
    # Compute 95% confidence interval
    ci = stats.t.interval(confidence=0.95, df=sample_size-1, loc=sample_mean, scale=standard_error)
    
    # Calculate w (half the width of the confidence interval)
    w = (ci[1] - sample_mean)
    
    # Calculate the scaled distance from the true mean to the sample mean
    distance_scaled = (true_mean - sample_mean) / w
    
    # Store the result
    distance_in_terms_of_w.append(distance_scaled)