# What's wrong with this interpretation of a 95% confidence interval?

Note: I asked a version of this as part of another question, but I'm re-asking it as a stand-alone question with more detail.

I've been trying to come up with more intuitive/less confusing ways to accurately explain what a given (say) 95% confidence interval (e.g. "the 95% CI runs between 2.5 and 3.7") actually means. I know that it does not mean "there is a 95% chance that the true value is within this interval" (because the true value is not a random variable) but that it is correct to say that, "out of all possible 95% CIs that could have been calculated from these data 95% of them will include the true value." However, based on my interpretation of the Central Limit Theorem and frequentism, I believe the following statement should also be valid interpretation of a 95% confidence interval that ranges between A and B:

"if it were true that the true mean was the the value we actually estimated, and we replicated our study a 100 times, estimating the mean each time, then 95% of those estimates would fall between A and B."

I've been told this is wrong, but I just want to understand why it's wrong.

Here's my logic. Let me know if and where there is a problem.

(First off, to avoid having to talk about t distributions let's just say that all sample sizes in this discussion are large enough that the distinction between normal and t distributions is irrelevant).

We want to estimate some parameter $$\mu$$ in a given population. The CTL says that if we draw a large number of random samples from this population and generate an estimate $$\hat \mu$$ in each of them, these estimates will form a normal curve, with a particular standard deviation, centered on the true value $$\mu$$. We therefore know, for example, that 95 percent of these estimates will be within 1.96 standard deviations of the true value. Of course, we only have one of these estimates, and we don't know how big the standard deviation of this sampling distribution actually is. However, we use the standard deviation $$\hat \sigma$$ of $$\hat \mu$$ in the data we actually collected as an unbiased estimator and thus estimate the standard error as $$\hat \sigma / \sqrt n$$.

Now, let's assume for the sake of argument that $$\mu=\hat \mu$$. (obviously this is unlikely to be the case, but it is analogous to the hypothetical we use when interpreting p values: "if the null hypothesis were true..."). Under this assumption the CLT tells us that if we again took repeated random samples, other estimates of $$\mu$$ from those samples will be distributed normally around $$\hat \mu$$ (because $$\mu=\hat \mu$$), meaning that 95% of those estimates will be within 1.96 standard deviations of $$\hat \mu$$. As noted above, we estimate the size of "one standard deviation" in the sampling distribution by the standard error. This allows us to can calculate the following

$$A=\hat \mu+1.96\times SE$$ $$B=\hat \mu-1.96\times SE$$

We can therefore say "if $$\mu= \hat \mu$$, and we replicated our study a large number of times times, estimating the $$\hat \mu$$ each time, then 95% of those estimates of would fall between A and B."

Of course, the formula I used was just the standard formula to calculate a 95% confidence interval around $$\hat \mu$$. So it seems like the sentence above should be a valid interpretation of a 95% confidence interval. If it's not, why not?

• You might find the graphical analysis of confidence interval construction at stats.stackexchange.com/a/426994/919 to be enlightening. Among other things, it shows conceptually that the CLT is irrelevant; normality is irrelevant; and lack of bias in estimates is irrelevant for understanding confidence intervals.
– whuber
Commented Jan 3, 2022 at 15:36

Your statement is imprecise in terms of true and estimated values, but not so much of the mean, but of the standard deviation.

if it were true that the true mean was the the value we actually estimated, and we replicated our study a 100 times, estimating the mean each time, then 95% of those estimates would fall between A and B.

Note that A and B depend on a single estimate of the standard deviation, namely the one from the original experiment. Your replications only estimate the mean, not standard deviations.

If your original experiment underestimated the SD, then replications will yield estimates of the mean that are contained in your original interval [A,B] in fewer than 95% of cases. If your original experiment overestimated the SD, your replicates will fall in the interval more often than 95%. And you will either under- or overestimate the SD with probability 1.

With whuber, the solution lies in keeping the CI calculation and a result from such a calculation (which is known as "a CI") strictly separate. A CI can only be interpreted as a particular result from a CI calculation. Yes, this is unintuitive. Unfortunately, there is simply no intuitive and correct interpretation of CIs.

• Ah, thank you, that does make sense. I had a hunch that the SD part of this was what was tripping me up, but wasn't sure. I guess this is why statisticians decided that the best way to describe these things was to just redefine the word "confidence" to mean something unintuitive and obscure, so that CIs seem intuitive to people who think the word means what it does in normal English (a statement about subjective belief). Commented Jan 3, 2022 at 15:00
• Unfortunately, there is simply no intuitive and correct interpretation of CIs – sounds true... Commented Jan 3, 2022 at 15:05
• "Unfortunately, there is simply no intuitive and correct interpretation of CIs." Intuition is subjective. For me the straight "in the long run, 95% CIs computed in this way from samples generated from the assumed model will contain the true parameter" is intuitive enough. Commented Jan 3, 2022 at 23:13

The big problem with your statement ["if it were true that the true mean was the value we actually estimated, and we replicated our study a 100 times, estimating the mean each time, then 95% of those estimates would fall between A and B."] is that we want to make an inference about the true mean, whereas you are discussing a hypothetical application in which the true mean has a different value.

Regarding confidence intervals, to calculate a confidence interval you have already assumed a statistical model, say with random variable X, observed sample x, and random confidence intervals C(X) = [L(X, U(X)]. In short, the lower and upper limits bounce around from sample to sample but 95% of the possible intervals contain the true mean. Post-sample, my confidence interval C(x) = [L(x), U(x)] either does or does not contain the true mean. The genius of Neyman was to use the word “confidence” to explicitly acknowledge a different meaning than would obtain were the word “probability” used in place of it.

Suppose I compute a 95% confidence interval for the true mean. Given the assumed model, we state 95% confidence that the CI produced by my sample contains the true mean only because the assumed process (i.e., the model and C(X)) produces correct confidence intervals 95% of the time.

Sure, I don’t know whether the observed CI is correct or not, but the 95% accuracy of the process that generated it is somewhat reassuring. Given the assumed model, either the observed CI contains the true mean or we have suffered a rare event [Note the past tense here.] In particular, our 95% confidence claim is neither a probability, nor a posterior probability, nor a belief. Note that there is nothing random left, the observed confidence interval either contains the true mean or it does not. By using the word confidence, we face up to that reality.

Had the degree of confidence been higher, say 99.9%, we would have been even more confident (because if it doesn’t contain the true mean then it means I have suffered extreme bad luck [note past tense again] that occurs only 0.1% of the time).

There is an important caveat regarding such confidence claims. Confidence would be undermined if it was known that the confidence interval procedure that I was using had poor conditional properties. (There are a number of contrived examples that demonstrate this.) In such circumstances, it is better to base confidence claims on appropriate conditional probabilities rather than on the unconditional probability.

If you want an authority for why confidences are not the same as probabilities, the text Cox and Hinkley (1974) p. 227-228 spells out why confidences cannot be manipulated as if they are probabilities.

• I don't like the wording "I am/we are 95% confident", because the 95% are a mathematical statement about a property of the CI under an idealised model; there is no necessary implication about my/your/our personal confidence. (Personally, in any real situation, I believe that the assumed model is not true and a true parameter value therefore does not exist, so for sure I'm not confident at all that this non-existing value is in the CI.) Commented Jan 3, 2022 at 23:19
• Good point about the idealised model, As Cox says, everything in statistics is provisional. I have edited my answer to say" Given the assumed model, we state 95% confidence that ..." I have changed the "I" to "we" regarding the claim as it is meant to include other (frequentist) statisticians using the same model. Other well-known frequentists are happy with the "we are 95% confident" format. Commented Jan 4, 2022 at 0:00
• For example, the (frequentist) statistician Kiefer's (1977) JASA article extended the confidence concept in a number of ways including to conditional confidence. He wrote, for example, "we are conditionally 96% confident that our decision is correct" p. 791. As you say, these confidences are not personal. Confidence claims are a consequence of the model selected and the sample only. Commented Jan 4, 2022 at 0:02