Misconceptions about the Meaning of Confidence Intervals [duplicate]

Question

NOTE

A few members have suggested the below question as a possible answer. Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

Neither the above question nor the responses address the question I have asked. I have given steps of deduction to start from the truth and arrive at a misconception. I would like to know why that deduction is incorrect.

Suppose we create a 95% CI from a sample in order to estimate the population mean. Let's say the 95% CI turned out to be [34.2, 36.7]

Under the Frequentists view, .

1) The population mean is a fixed parameter

2) If we were to take very large number of random samples of the same size and create 95% CI from every sample, 95% of those CIs would contain the true population mean.

The below is said to be a misconception.

3) There is a 95% probability for the true population mean to be present in the above 95% CI which is [34.2, 36.7]

Two reasons and justification given by the Frequentists are below.

a) "Once a 95% CI is calculated, it either contains the population mean or not. No more probability (doubt) whether the CI contains the population mean or not".

I don't quite understand this argument. This is obviously true if the population mean is known. But if the true population mean is unknown, then the CI only has a chance (probability) to contain the population mean.

b) This is a proof by contradiction to prove that above statement 3 is wrong

Suppose there is a 95% probability for the CI [34.2, 36.7] to contain the population mean. If we repeat the sampling process many times, we can get a sample with a CI that is outside the initial CI. Eg [36.9, 37.3]

Now it not possible for each of these two non-overlapping CI ranges to have a 95% probability each to contain the true population mean.

But, I thought we can start with Statement 2 and deduce Statement 3

Let's start like this.

We take a random sample, create the 95% CI and write it on a piece of paper.

We repeat this process a lot of times.

Now we have so many pieces of paper.

The above statement 2 deduces the argument below

4) The CIs made from 95% the samples contain the true population mean.

==> 95% of these pieces of paper contain the true population mean.

==> any single piece of paper has a 95% probability to contain the true population mean.

==> any single CI has a 95% probability to contain the true population mean.

This is essentially statement 3 above which is the "misconception".

So why is this reasoning wrong?

UPDATE 1

Based on the feedback from @Mathemagician777, is the below statement true or a misconception?

3.5) "The interval [34.2, 36.7] has a 95% probability to capture the true population mean"

UPDATE 2

Two members have suggested the below question as a possible answer. Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

Neither the above question nor the responses address the question I have asked. I have given steps of deduction to start from the truth and arrive at a misconception. I would like to know why that deduction is incorrect.

UPDATE 3

A common response I have seen is that "the population mean is fixed, therefore, it is incorrect to argue that a particular CI has a probability to contain the population mean".

Let me counter argue with an example from a die roll.

• Person A rolls a die and hides the outcome from Person B. Now the outcome is fixed just like the population mean. The die is now returned and plays no part any longer in this experiment.
• Person B makes a series of guesses, each time with a number between 1 to 6.
• Each of these guesses ( Confidence Intervals) has a 1/6th probability contain the outcome of the initial die roll which is hidden from Person B.
• As an example, Person B says "I guess 4" and that guess has a 1/6th probability to correctly contain the initial die roll outcome.

This example shows that even though the die roll outcome was fixed but hidden, probabilities still exist about the ability of the guesses to guess (contain) the hidden outcome.

In the above example, the die roll outcome is similar to the fixed population mean, and each guess is similar to the 95% CI made from a random sample.

So, why can't a particular 95% CI have a 95% probability of containing the population mean?

Answer 1

You are mixing up the meaning behind statement 2 and 3.

In statement 2, you correctly put the 95% uncertainty around your confidence interval. While in statement 3, the uncertainty is about your population parameter (mean).

An example: Suppose you study the shoe size of all Belgian people (= population) by taking a sample of 1000 persons and calculate a mean for this. The mean shoe size of the population is unknown then, but it is not gonna vary. If you had the tools and could perform a census, you could get the exact estimate.

However, there are an infinite different ways to sample 1000 persons from this population, and depending on your sample, you get a different estimate and thus a different confidence interval!

So the correct statements are the ones that imply that the 95% uncertainty is about the confidence interval: you either collect a sample and make a confidence interval based on this sample that contains the population parameter, or you don't, but the population parameter will not change!

Saying something like 'There is 95% uncertainty that the population parameter lies in the 95% confidence interval' seems to imply that the confidence interval is fixed, and the population parameter varies and might be situated in the confidence interval, while it is exactly the other way around!

Where goes the reasoning in your example wrong: In the very last sentence you say any single CI has a 95% probability to contain the true population mean. which is correctly about the confidence interval, but then you say this is equivalent to statement 3, which is not true! Statement 3 puts the uncertainty on the population parameter, and thus this final implication is not true: you have not implied statement 3 with statement 2 (luckily ;) ).

Answer 2

It is very natural to consider things which are unmeasured as being random. In this case, you think of the population mean as being the random thing. The problem is the "so what?" - in other words, how does your experiment actually update what you believe about this parameter?

Bayesian statistics give us the quantitative framework to update belief based on data, but the problem is that all priors are subjective - even the noninformative ones - which is antithetical to science (according to some). So Bayesians can make claims about the probability of capturing a population mean within a credible interval.

Frequentists on the other hand strive for an objective approach. So what you state in 3a is absolutely true - the confidence interval is not random because you collected the data, and the population mean is not random - it is a specific value we just don't have all the $n$ to know what it is. What is random is the counterfactual - rerunning the experiment indefinitely.

It is true that if we ran infinite replicates of a given design, we can claim as frequentists that 95% of those intervals contain the true mean and this is the frequentist's definition of probability. However we are not in fact drawing inference from many studies, we have but a single study.

Answer 3

Answer to the Update 3 version of your question (I follow the convention of using uppercase for random variables and lower case for particular random sample outcomes).

• “Person A rolls a die and hides the outcome from Person B. Now the outcome is fixed just like the population mean. The die is now returned and plays no part any longer in this experiment.”

Statistician: Assuming a fair die and a fair rolling process, all values of D are equally likely before the die roll. As it happens, the roll outcome is d = 2, but person B does not know this.

• “Person B makes a series of guesses, each time with a number between 1 to 6.

• Each of these guesses ( Confidence Intervals) has a 1/6th probability contain the outcome of the initial die roll which is hidden from Person B.”

Statistician: Now you have introduced information about a random sequence of Person B’s guesses, and we could try to model B’s guess generating process, as a rv G with possible sample outcomes the same as D. Before guessing commences, but after the die roll, P(guess is correct) = P(G = 2) = 1/6. Person B announces first guess is g = 4. This is a wrong guess, so it cannot have a 1/6 probability of being correct. [Paralleling Neyman's use of the word 'confidence', up until d = 2 is revealed to Person B, B might state 100/6% confidence that g = 4 is correct.] The situation is similar to a single observed confidence interval: it either covers or does not cover the unknown parameter value.

Note that in the frequentist approach outlined above, the modelling is about the random processes, rather than trying to model a person's uncertainty about a particular outcome. The latter approach is employed by the Bayesian method which would output a posterior probability that g = 4 is correct rather than a confidence.