67
$\begingroup$

I have found discordant information on the question: "If one constructs a 95% confidence interval (CI) of a difference in means or a difference in proportions, are all values within the CI equally likely? Or, is the point estimate the most likely, with values near the "tails" of the CI less likely than those in the middle of the CI?

For instance, if a randomized clinical trial report states that the relative risk of mortality with a particular treatment is 1.06 (95% CI 0.96 to 1.18), is the likelihood of 0.96 being the correct value the same as 1.06?

I found many references to this concept online, but the following two examples reflect the uncertainty therein:

  1. Lisa Sullivan's module about Confidence Intervals states:

    The confidence intervals for the difference in means provide a range of likely values for ($μ_1-μ_2$). It is important to note that all values in the confidence interval are equally likely estimates of the true value of ($μ_1-μ_2$).

  2. This blogpost, titled Within the Margin of Error, states:

    What I have in mind is misunderstanding about “margin of error” that treats all points within the confidence interval as equally likely, as if the central limit theorem implied a bounded uniform distribution instead of a t distribution. [...]
    The thing that talk about “margin of error” misses is that possibilities that are close to the point estimate are much more likely than possibilities that are at the edge of the margin".

These seem contradictory, so which is correct?

$\endgroup$
7
  • 9
    $\begingroup$ I wonder if there is confusion somewhere with the related concept that p-values are uniformly distributed under the null hypothesis.. $\endgroup$ Commented Oct 19, 2012 at 19:01
  • 5
    $\begingroup$ The first quotation is an aberrant slip in an otherwise accurate account of confidence intervals. The second quotation is from an account that, to put it nicely, is a sloppy mess: it's full of statements that are vague, incorrect, or can only be interpreted in a Bayesian sense. But both quotations are wrong! $\endgroup$
    – whuber
    Commented Oct 19, 2012 at 19:18
  • 1
    $\begingroup$ @Michael One example of sloppiness is a solecism like asserting the CLT implies an "infinite number of repeated estimates of [the population] mean will still follow a normal distribution." One does not have to be wrong in order to communicate ideas simply to a nontechnical audience. $\endgroup$
    – whuber
    Commented Oct 19, 2012 at 19:30
  • 2
    $\begingroup$ @whuber, I consider the sentence you cite only a minor sin. The main error is that CLT does not involve t distribution. $\endgroup$
    – glassy
    Commented Jan 15, 2013 at 8:39
  • 1
    $\begingroup$ @user1205901-СлаваУкраїні - I was disappointed with that post from Gelman and find much more of interest on the present page. $\endgroup$
    – rolando2
    Commented Apr 23 at 20:33

7 Answers 7

30
$\begingroup$

One question that needs to be answered is what does "likely" mean in this context?

If it means probability (as it is sometimes used as a synonym of) and we are using strict frequentist definitions then the true parameter value is a single value that does not change, so the probability (likelihood) of that point is 100% and all other values are 0%. So almost all are equally likely at 0%, but if the interval contains the true value, then it is different from the others.

If we use a Bayesian approach then the CI (Credible Interval) comes from the posterior distribution and you can compare the likelihood at the different points within the interval. Unless the posterior is perfectly uniform within the interval (theoretically possible I guess, but that would be a strange circumstance) then the values have different likelihoods.

If we use likely to be similar to confidence then think about it this way: Compute a 95% confidence interval, a 90% confidence interval, and an 85% confidence interval. We would be 5% confident that the true value lies in the region inside of the 95% interval but outside of the 90% interval, we could say that the true value is 5% likely to fall in that region. The same is true for the region that is inside the 90% interval but outside the 85% interval. So if every value is equally likely, then the size of the above 2 regions would need to be exactly the same and the same would hold true for the region inside a 10% confidence interval but outside a 5% confidence interval. None of the standard distributions that intervals are constructed using have this property (except special cases with 1 draw from a uniform).

You could further prove this to yourself by simulating a large number of datasets from known populations, computing the confidence interval of interest, then comparing how often the true parameter is closer to the point estimate than to each of the end points.

$\endgroup$
7
  • 3
    $\begingroup$ Likelihood is what this question needs in the answer, not probability, either frequentist or Bayesian. Likelihood provides exactly the answer, the others can only do so with some twisting and stretching. $\endgroup$ Commented Oct 20, 2012 at 7:21
  • 1
    $\begingroup$ @Greg I like your explanation. Just to be clear, your argument supports the notion that the values at the "tails" of the 95% CI are less likely (less probable) than those closer to the point estimate, correct? Thanks for your response. $\endgroup$
    – pmgjones
    Commented Oct 21, 2012 at 14:15
  • 1
    $\begingroup$ @pmgjones less probable, NO, see 2nd paragraph. Less likely in the context of the 4th paragraph, Yes. $\endgroup$
    – Greg Snow
    Commented Oct 22, 2012 at 17:35
  • 2
    $\begingroup$ @GregSnow Your 2nd paragraph says almost exactly that the probability of the true parameter being the true parameter is 100%. Do you really believe that this tautology is what "strict frequentist definitions" have to offer? $\endgroup$
    – rolando2
    Commented Jan 26, 2013 at 4:13
  • 2
    $\begingroup$ @rolando2, I think that frequentist statistics has much to offer, I was just eliminating the common misstatements that imply the true value changes and is sometimes out of the interval and sometimes inside the interval (and sometimes closer to the boundaries and sometimes closer to the center). The later paragraphs then get at the more true feel for the ideas. $\endgroup$
    – Greg Snow
    Commented Jan 26, 2013 at 16:11
25
$\begingroup$

Suppose someone told me that I should place equal trust in all values within a CI95 as potential indicators of the population value. (I'm deliberately avoiding the terms "likely" and "probable.") What's special about 95? Nothing: to be consistent I would also have to place equal trust in all values within a CI96, a CI97, ... and a CI99.9999999. As the CI's coverage approached its limit, virtually all real numbers would have to be included. The preposterousness of this conclusion would lead me to reject the initial claim.

$\endgroup$
1
  • 4
    $\begingroup$ This is a great answer! I should have thought of the effect of approaching extremes of possible CIs. Thanks for writing this! $\endgroup$
    – pmgjones
    Commented Jan 26, 2013 at 13:45
21
$\begingroup$

This is a great question! There is a mathematical concept called likelihood that will help you understand the issues. Fisher invented likelihood but considered it to be somewhat less desirable than probability, but likelihood turns out to be more 'primitive' than probability and Ian Hacking (1965) considered it to be axiomatic in that it is not provable. Likelihood underpins probability rather then the reverse.

Hacking, 1965. Logic of Statistical Inference.

Likelihood is not given the attention that it should have in standard textbooks of statistics, for no good reason. It differs from probability in having almost exactly the properties that one would expect, and likelihood functions and intervals are very useful for inference. Perhaps likelihood is not liked by some statisticians because there is sometimes no 'proper' way to derive the relevant likelihood functions. However, in many cases the likelihood functions are obvious and well defined. A study of likelihoods for inference should probably start with Richard Royall's small and easy to understand book called Statistical Evidence: a Likelihood Paradigm.

The answer to your question is that no, the points within any interval do not all have the same likelihood. Those at the edges of a confidence interval usually have lower likelihoods than others towards the centre of the interval. Of course, the conventional confidence interval tells you nothing directly about the parameter relevant to the particular experiment. Neyman's confidence intervals are 'global' in that they are designed to have long-run properties rather than 'local' properties relevant to the experiment in hand. (Happily good long-run performance can be interpreted in the local, but that is an intellectual shortcut rather than a mathematical reality.) Likelihood intervals—in the cases where they can be constructed—directly reflect the likelihood relating the experiment in hand. There is less about likelihood intervals that is confusing than is the case for confidence intervals, in my opinion, and they have more utility than might be expected from their almost complete disuse!

$\endgroup$
11
  • 2
    $\begingroup$ @suncoolsu It is not necessary that the interval in question be a likelihood interval for the statement to be true. The interval only has to span the most likely estimate so that the interval bounds are each less likely than a point within the interval. Any ordinary confidence interval will satisfy that requirement. $\endgroup$ Commented Oct 21, 2012 at 6:48
  • 2
    $\begingroup$ @pmjones A 95% CI DOEST NOT tell you if the values towards the margins of the CI are closer to the truth than the values in the middle. CIs make statements about repeated sampling from the population. In the long run (i.e., after repeated sampling), 95% of such CIs, which are constructed for each sample, will cover the true value. Therefore, there are two key observations 1) One cannot say anything about the true value for a given CI 2) CIs do not tell you anything about the observed data, which is a usual Bayesian criticism. $\endgroup$
    – suncoolsu
    Commented Oct 21, 2012 at 19:09
  • 1
    $\begingroup$ @MichaelLew Likelihood principle is useful, but I was saying that (quoting LW) "Indeed, all of frequentist inference violates LP, so if we adhered to LP we would have to abandon frequentist inference." Because CI is a frequentist idea, it violates LP (which you say is fundamental). $\endgroup$
    – suncoolsu
    Commented Oct 21, 2012 at 19:12
  • 1
    $\begingroup$ @suncollsu The question is not whether a confidence interval alone and without any other statistical considerations tells anything about the likelihood of parameter values within itself. It is about the likelihood of parameter values within the interval. The likelihood function answers the question, and that answer is correct even if the confidence interval violates the likelihood principle. (Read my previous comment again. You seem to have ignored its content entirely.) $\endgroup$ Commented Oct 21, 2012 at 20:13
  • 2
    $\begingroup$ @rolando2 Neyman's 95% confidence intervals are designed so that the method contains the true parameter on 95% of occasions that the method is used. Strictly speaking the confidence attaches to the method and not to any individual interval and so the individual interval does not tell you anything about the state of the world in that particular experiment. See my answer to this question for more detail: stats.stackexchange.com/questions/8844/… $\endgroup$ Commented Jan 27, 2013 at 20:05
2
$\begingroup$

Let's start with the definition of a confidence interval. If I say that a 95% confidence interval goes from this to that I mean that statements of that nature will be true about 95% of the time and false about 5% of the time. I do not necessarily mean that I am 95% confident about this particular statement. A 90% confidence interval will be narrower and an 80% narrower still. Therefore, when wondering what the true value is, I have less credence in values as they get closer and closer to the edge of any particular confidence interval.

Note that all of the above is qualitative, especially "credence". (I avoided the term "confidence" or "likelihood" in that statement because they carry mathematical baggage that may differ from our intuitive baggage.) Bayesian approaches would rephrase your question to something that has a quantitative answer but I don't want to open that can of worms here.

Box, Hunter & Hunter's classic text ("Statistics for Experimenters", Wiley, 1978) may also help. See "Sets of Confidence Intervals" on pp 113, ff.

$\endgroup$
1
  • $\begingroup$ Since we are dealing partly in concepts and partly in semantics, I'll point out that in your second sentence you've said "...statements of that nature will be true..." without specifying what statements would be true. $\endgroup$
    – rolando2
    Commented Jan 26, 2013 at 23:40
2
$\begingroup$
  1. I recommend reading the other answers, I thought the answer by Greg Snow was excellent, and the discussion by Michael Lew on likelihood very interesting.
  2. I want to encourage people to make use of tools like R and Python to explore (counter) examples and develop their intuition. This is what we will do in this post, essentially the simulation suggested at the end of Greg's answer.

The plan

In Python, we will create a normal distribution, where the true mean is $\mu=0$ and the standard deviation is $\sigma=1$.

  • We will repeatedly take random samples from this distribution and calculate 95% confidence intervals for the mean. To do this we will use the t-distribution (this method is commonly taught).
  • These intervals are symmetric, and the midpoint is the sample mean $\bar{x}$.
  • The confidence intervals we generate will have different widths, depending on the sample (in particular the sample standard deviation).
  • For any given sample say the width of the confidence interval is $2w$, so the distance from the midpoint(the sample mean $\bar{x}$) to the end of the interval is $w$.
  • We will compute the distance from the interval midpoint to the true mean, and then divide by the width $w$, to put this distance in terms of $w$. Call this $d$. That is $d = \frac{(\bar{x} - \mu)}{w}$. We do this normalisation because the confidence intervals could have different widths.
  • Finally we will plot the distribution of these $d$s.
  • We will show that if we repeat the simulation many times, that the true mean is more frequently closer to the midpoints of the confidence intervals than it is to either of the endpoints.

Simulation Plots

We plot first a histogram of the distances $d$, and then the cumulative distribution of the (absolute) distance $|d|$

Histogram showing distribution of the ds

Cumulative frequencies of the absolute values of the ds

Highlights

  • We see from the simulation that values of $d$ around 0 occur most frequently (that is, the true mean is close to the midpoint), and larger distances $d$ occur more rarely.
  • When the absolute distance is less than $\frac{w}{2}$, it means the true mean is closer to the interval midpoint than either endpoint. In the cumulative plot, we can see that happens about 70% of the time.
  • Side note: The bars shaded red in the histogram are when the distance is greater than $w$, this is when the true mean is not covered by the interval. In this case it happens about 5% of the time (recall we were generating 95% confidence intervals).

Code

The core part of the Python code (not including the plotting) is copied below.

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

# Parameters
n_simulations = 10000
sample_size = 30
true_mean = 0
std_dev = 1

# Store the distance from the true mean in terms of w for each simulation
distance_in_terms_of_w = []

# Run simulations
for _ in range(n_simulations):
    # Generate sample
    sample = np.random.normal(loc=true_mean, scale=std_dev, size=sample_size)
    
    # Calculate sample mean and standard error
    sample_mean = np.mean(sample)
    standard_error = stats.sem(sample)
    
    # Compute 95% confidence interval
    ci = stats.t.interval(confidence=0.95, df=sample_size-1, loc=sample_mean, scale=standard_error)
    
    # Calculate w (half the width of the confidence interval)
    w = (ci[1] - sample_mean)
    
    # Calculate the scaled distance from the true mean to the sample mean
    distance_scaled = (true_mean - sample_mean) / w
    
    # Store the result
    distance_in_terms_of_w.append(distance_scaled)
$\endgroup$
2
$\begingroup$

Although the phrase “all values within the CI equally likely” is unclear in its meaning, it would seem to imply that our confidences in equal-length segments of the CI should be the same irrespective of where in the CI they are located. However this implication is false. To demonstrate this, consider the example with two normally distributed random variables with known variances, and CI based on:

In summary, for this example, confidence per unit length near the outer boundaries of the CI is much less than it is near the centre of the CI.

$\endgroup$
1
$\begingroup$

Perhaps this will help in thinking about this issue. Imagine two ways to define a 90% confidence region. The standard 90% CI (the central case) versus the 90% confidence region constructed as the standard 95% CI minus the interval included in a standard 5% CI.

$\endgroup$
4
  • 1
    $\begingroup$ Yes, a discontinuous interval can be a frequentist confidence interval, and even if it omits the most likely region of parameter space. But such an interval is well outside the convention where the confidence intervals are most often chosen to be shortest and or central. For your discontinuous interval an answer to the original question would need more words, but it would still be the case that the parameter values within the interval would have varying likelihoods. $\endgroup$ Commented Apr 24 at 2:47
  • $\begingroup$ My hope is that my answer is thought-provoking (rather than complete), and that readers would understand the indirect relevance for the question asked. $\endgroup$ Commented Apr 24 at 3:04
  • 2
    $\begingroup$ Well, it was thought provoking to me, but I think it needs a bit more detail to be accessible. $\endgroup$ Commented Apr 24 at 3:09
  • $\begingroup$ @MichaelLew I have added a more-direct answer to the question, and will probably delete this first one. $\endgroup$ Commented Jul 10 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.