# Why does a 95% CI not imply a 95% chance of containing the mean?

It seems that through various related questions here, there is consensus that the "95%" part of what we call a "95% confidence interval" refers to the fact that if we were to exactly replicate our sampling and CI-computation procedures many times, 95% of thusly computed CIs would contain the population mean. It also seems to be the consensus that this definition does not permit one to conclude from a single 95%CI that there is a 95% chance that the mean falls somewhere within the CI. However, I don't understand how the former doesn't imply the latter insofar as, having imagined many CIs 95% of which contain the population mean, shouldn't our uncertainty (with regards to whether our actually-computed CI contains the population mean or not) force us to use the base-rate of the imagined cases (95%) as our estimate of the probability that our actual case contains the CI?

I've seen posts argue along the lines of "the actually-computed CI either contains the population mean or it doesn't, so its probability is either 1 or 0", but this seems to imply a strange definition of probability that is dependent on unknown states (i.e. a friend flips fair coin, hides the result, and I am disallowed from saying there is a 50% chance that it's heads).

Surely I'm wrong, but I don't see where my logic has gone awry...

-
By "chance", do you mean "probability" in the technical frequentist sense, or in the Bayesian sense of subjective plausibility? In the frequentist sense, only events of random experiments have a probability. Looking at three given (fixed) numbers (true mean, calculated CI bounds) to determine their order (true mean contained in CI?) is not a random experiment. This is also why the probability-part of "the actually-computed CI either contains the population mean or it doesn't, so its probability is either 1 or 0" is wrong as well. A frequentist probability model just doesn't apply in that case. –  caracal Apr 14 '12 at 12:38
It depends on how you treat the theoretical mean. If it is random variable then you can say about probability that it falls into some interval. If it is constant, you cannot. That is the most simple explanation, which closed this issue for me personally. –  mpiktas Apr 14 '12 at 17:59
Incidentally, I came across this talk, from Thaddeus Tarpey: All models are right… most are useless. He discussed the question of the probability that a 95 % confidence interval contains $\mu$ (p. 81 ff.)? –  chl Apr 14 '12 at 21:25
@Nesp: I do not think there is any issue with the statement "It's probability is either zero or one" in reference to the (posterior) probability that a CI contains a (fixed) parameter. (This does not even really rely on any frequentist interpretation of probability!). It also does not rely on "unknown states". Such a statement refers precisely to the situation in which one is handed a CI based on a particular sample. It is a simple mathematical exercise to show that any such probability is trivial, i.e., takes values in $\{0,1\}$. –  cardinal Apr 15 '12 at 16:37
@cardinal Yes, as you say, I don't see an issue either. However (and maybe there my english failed) I intended to say that it can be misleading if not explained properly (e.g. using Bayes theorem) :-). –  Néstor Apr 15 '12 at 18:50

Part of the issue is that the frequentist definition of a probability doesn't allow a nontrivial probability to be applied to the outcome of a particular experiment, but only to some fictitious population of experiments from which this particular experiment can be considered a sample. The definition of a CI is confusing as it is a statement about this (usually) fictitious population of experiments, rather than about the particular data collected in the instance at hand. So part of the issue is one of the definition of a probability: The idea of the true value lying within a particular interval with probability 95% is inconsistent with a frequentist framework.

Another aspect of the issue is that the calculation of the frequentist confidence doesn't use all of the information contained in the particular sample relevant to bounding the true value of the statistic. My question "Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals" discusses a paper by Edwin Jaynes which has some really good examples that really highlight the difference between confidence intervals and credible intervals. One that is particularly relevant to this discussion is Example 5, which discusses the difference between a credible and a confidence interval for estimating the parameter of a truncated exponential distribution (for a problem in industrial quality control). In the example he gives, there is enough information in the sample to be certain that the true value of the parameter lies nowhere in a properly constructed 90% confidence interval!

This may seem shocking to some, but the reason for this result is that confidence intervals and credible intervals are answers to two different questions, from two different interpretations of probability.

The confidence interval is the answer to the request: "Give me an interval that will bracket the true value of the parameter in $100p$% of the instances of an experiment that is repeated a large number of times." The credible interval is an answer to the request: "Give me an interval that brackets the true value with probability $p$ given the particular sample I've actually observed." To be able to answer the latter request, we must first adopt either (a) a new concept of the data generating process or (b) a different concept of the definition of probability itself.

The main reason that any particular 95% confidence interval does not imply a 95% chance of containing the mean is because the confidence interval is an answer to a different question, so it is only the right answer when the answer to the two questions happens to have the same numerical solution.

In short, credible and confidence intervals answer different questions from different perspectives; both are useful, but you need to choose the right interval for the question you actually want to ask. If you want an interval that admits an interpretation of a 95% (posterior) probability of containing the true value, then choose a credible interval (and, with it, the attendant conceptualization of probability), not a confidence interval. The thing you ought not to do is to adopt a different definition of probability in the interpretation than that used in the analysis.

Thanks to @cardinal for his refinements!

-
The confidence interval is the answer to the question "give me an interval that will bracket the true value of the statistic with probability p if the experiment is repeated a large number of times". The credible interval is an answer to the question "give me an interval that brackets the true value with probability p". First of all, the statement regarding a frequentist interpretation of probability leaves something to be desired. Perhaps, the issue lies in the use of the word probability in that sentence. Second, I find the credible interval "definition" to be a little too simplistic... –  cardinal Apr 14 '12 at 17:38
...and slightly misleading considering the characterization you give to a CI. In a related vein, the closing sentence has the same issue: If you want an interval that contains the true value 95% of the time, then choose a credible interval, not a confidence interval. The colloquial use of "contains the true value 95% of the time" is a bit imprecise and leaves the wrong impression. Indeed, I can make a convincing argument (I believe) that such wording is much closer to being the definition of a CI. –  cardinal Apr 14 '12 at 17:42
Request: It would be helpful for the downvoter to this answer to express their opinion/reasons in the comments. While this question is a bit more likely than most to lead to extended discussion, it is still useful to provide constructive feedback to answerers; that is one of the easiest ways to help improve the overall content of the site. Cheers. –  cardinal Apr 14 '12 at 19:06
downvoterS! I agree, the feedback would be helpful. –  Dikran Marsupial Apr 14 '12 at 19:09
Dikran, yes, I agree. That was part of what I was trying to draw out a little bit more in the edits. A radical frequentist (which I am certainly not) might state it provocatively as: "A CI is conservative in that I design the interval beforehand such that no matter what particular data I happen to observe, the parameter will be captured in the interval 95% of the time. A credible interval arises from saying 'Oops, someone just threw some data in my lap. What's the probability the interval I construct from that data contains the true parameter?'" That is a bit unfair in the latter case... –  cardinal Apr 14 '12 at 20:20

Probabilities related to theoretical long run events, that even work if you run the events in the long run, just don't relate to a single event after it's done. And the running of an experiment and calculation of the CI is just such an event.

You wanted to compare it to the probability of a hidden coin being heads.. maybe I can work with that. Let's see if I can completely screw this up sticking to your analogy.

When you run your experiment and collect your data then you've got something similar to the actual flip of the coin. The process of the experiment is like the process of the coin flipping in that it generates $\mu$ or it doesn't just like the coin is heads or it's not. Once you flip the coin, whether you see it or not, there is no probability that it's heads, it's either heads or it's not. Now suppose you call heads. That's what calculating the CI is. Because you can't ever reveal the coin (your analogy to an experiment would vanish). Either you're right or you're wrong, that's it. Does it's current state have any relation to the probability of it coming up heads on the next flip, or that I could have predicted what it is? No. The process by which the head is produced has a 0.5 probability of producing them but it does not mean that a head that already exists has a 0.5 probability of being. Once you calculate your CI there is no probability that it captures $\mu$, it either does or it doesn't—you've already flipped the coin.

OK, I think I've tortured that enough. The critical point is really that your analogy is misguided. You can never reveal the coin; you can only call heads or tails based on assumptions about coins (experiments). You might want to make a bet afterwards on your heads or tails being correct but you can't ever collect on it.

Probably the thing that makes the CI confusing is it's name. It's a range of values that either do or don't contain $\mu$. We think they likely contain $\mu$ but the probability of that isn't the same as the process that went into developing it. The 95% part of the 95% CI name is just about the process. You can calculate a range that you believe afterwards contains $\mu$ at some probability level but that's a different calculation and not a CI.

It's better to think of the name 95% CI as a designation of a kind of measurement of a range of values that you think plausibly contain $\mu$ and separate the 95% from that plausibility. We could call it the Jennifer CI while the 99% CI is the Wendy CI. That might actually be better. Then, afterwards we can say that we believe $\mu$ is likely to be in the range of values and no one would get stuck saying that there is a Wendy probability that we've captured $\mu$. If you'd like a different designation I think you should probably feel free to get rid of the "confidence" part of CI as well (but it is an interval).

-
To be fair enough this reply seems ok, but I'll love to see a formal (mathematical) description of it. With formal, I mean converting it to events. I'll explain my point: I remember being very confused with $p$ values at the start. Somewhere I read that "what $p$ values actually calculate are the probability of the data given that the null hypothesis, $H_0$, is true". When I related this with Bayes theorem, all made so much sense that now I can explain it to everyone (i.e. that one calculates $p(D|H_0)$). However, I'm (ironically) not that confident... –  Néstor Apr 14 '12 at 17:04
...(continued) with confidence intervals: is there a way to express what you said in terms of knowledge? In freq. stats. one usually calculates a point estimate, $\hat{\mu}$, with some method (e.g., MLE). Is there a way to write $P(L_1(\hat{\mu})<\mu<L_2(\hat{mu})|D)$ (e.g. with a bayesian central posterior interval, with $\mu$ the "true mean") as a function of $P(L_1'<\bar{X}-\mu<L_2')=\alpha$ (i.e. what the $\alpha$% of confidence intervals really is), as when you can express $p(H_0|D)$ as a function of $p(D|H_0)$? Intuitively I always have thought that it can be done, but never done it. –  Néstor Apr 14 '12 at 17:13
Sometimes being able to delete comments has its drawbacks. I couldn't keep up with the rapid changes, in this instance! –  cardinal Apr 14 '12 at 18:57
"If you don't calculate your confidence interval you've got something similar to the hidden coin and it has a 95% probability of containing mu just like the coin has a 50% probability of being heads." -- I think you got the analogy wrong here. "Calculating the CI" doesn't correspond to revealing the coin, it corresponds to calling "Heads" or "Tails", at which point you still have a 50-50 chance of being right. Revealing the coin corresponds to *seeing the population value of $\mu$, at which point you can answer the question of whether it's in the "called" interval. The OP's puzzle remains. –  Glen_b Aug 30 at 8:02
Yeah, I always hated that paragraph and never got back to fixing it or deleting it. I think I may have actually fixed it. –  John Aug 30 at 15:06

Formal, explicit ideas about arguments, inference and logic originated, within the Western tradition, with Aristotle. Aristotle wrote about these topics in several different works (including one called the Topics ;-) ). However, the most basic single principle is The Law of Non-contradiction, which can be found in various places, including Metaphysics book IV, chapters 3 & 4. A typical formulation is: " ...it is impossible for anything at the same time to be and not to be [in the same sense]" (1006 a 1). Its importance is stated slightly earlier, " ...this is naturally the starting-point even for all the other axioms" (1005 b 30). Forgive me for waxing philosophical, but this question by its nature has philosophical content that cannot simply be pushed aside for convenience.

Consider this thought-experiment: Alex flips a coin, catches it and turns it over onto his forearm with his hand covering the side facing up. Bob was standing in just the right position; he briefly saw the coin in Alex's hand, and thus can deduce which side is facing up now. However, Carlos did not see the coin--he wasn't in the right spot. At this point, Alex asks them what the probability is that the coin shows heads. Carlos suggests that the probability is .5, as that is the long-run frequency of heads. Bob disagrees, he confidently asserts that the probability is nothing else but exactly 0.

Now, who is right? It is possible, of course, that Bob mis-saw and is incorrect (let us assume that he did not mis-see). Nonetheless, you cannot hold that both are right and hold to the law of non-contradiction. (I suppose that if you don't believe in the law of non-contradiction, you could think they're both right, or some other such formulation.) Now imagine a similar case, but without Bob present, could Carlos' suggestion be more right (eh?) without Bob around, since no one saw the coin? The application of the law of non-contradiction is not quite as clear in this case, but I think it is obvious that the parts of the situation that seem to be important are held constant from the former to the latter. There have been many attempts to define probability, and in the future there may still yet be many more, but a definition of probability as a function of who happens to be standing around and where they happen to be positioned has little appeal. At any rate (guessing by your use of the phrase "confidence interval"), we are working within the Frequentist approach, and therein whether anyone knows the true state of the coin is irrelevant. It is not a random variable--it is a realized value and either it shows heads, or it shows tails.

As @John notes, the state of a coin may not at first seem similar to the question of whether a confidence interval covers the true mean. However, instead of a coin, we can understand this abstractly as a realized value drawn from a Bernoulli distribution with parameter $p$. In the coin situation, $p=.5$, whereas for a 95% CI, $p=.95$. What's important to realize in making the connection is that the important part of the metaphor isn't the $p$ that governs the situation, but rather that the flipped coin or the calculated CI is a realized value, not a random variable.

It is important for me to note at this point that all of this is the case within a Frequentist conception of probability. The Bayesian perspective does not violate the law of non-contradiction, it simply starts from different metaphysical assumptions about the nature of reality (more specifically about probability). Others on CV are much better versed in the Bayesian perspective than I am, and perhaps they may explain why the assumptions behind your question do not apply within the Bayesian approach, and that in fact, there may well be a 95% probability of the mean lying within a 95% credible interval, under certain conditions including (among others) that the prior used was accurate (see the comment by @DikranMarsupial below). However, I think all would agree, that once you state you are working within the Frequentist approach, it cannot be the case that the probability of the true mean lying within any particular 95% CI is .95.

-
What do you think of the following rewording of the last sentence? However, I think all would agree, that once you state you are working within the Frequentist approach, it cannot be the case that the probability of the true mean lying within any particular 95% CI is .95. –  cardinal Apr 14 '12 at 18:26
@cardinal, that seems like a good suggestion. Edited. –  gung Apr 14 '12 at 18:34
Under the Bayesian approach it isn't true that there is actually a 95% probability that the true value lies in a 95% credible interval. It would be more correct to say that given a particular prior distribution for the value of the statistic (representing our initial state of knowledge) then having observed the data we have a posterior distribution representing out updated state of knowledge, which gives us an interval where we are 95% sure that the true value lies. This will only be accurate if our prior is accurate (and other assumptions such as the form of the likelihood). –  Dikran Marsupial Apr 14 '12 at 19:56
@DikranMarsupial, thanks for the note. That's a bit of a mouthful. I edited my answer to make it more consistent with your suggestion, but did not copy it in toto. Let me know if further edits are appropriate. –  gung Apr 14 '12 at 20:20
Essentially the Bayesian approach is best interpreted as a statement of your state of knowledge regarding the parameter of interest (see cardinal, I am learning ;o), but doesn't guarantee that that state of knowledge is correct unless all of the assumptions are correct. I enjoyed the philosphical discussion, I shall have to remember the law of non-contradiction for the next time is discuss fuzzy logic ;o) –  Dikran Marsupial Apr 14 '12 at 20:48

I'm surprised that no one has brought up Berger's example of an essentially useless 75% confidence interval described in the second chapter of "The Likelihood Principle". The details can be found in the original text (which is available for free on Project Euclid): what is essential about the example is that it describes, unambiguously, a situation in which you know with absolute certainty the value of an ostensibly unknown parameter after observing data, but you would assert that you have only 75% confidence that your interval contains the true value. Working through the details of that example was what enabled me to understand the entire logic of constructing confidence intervals.

-
In a frequentist setting, one would not "assert that you have only 75% confidence that your interval contains the true value" in reference to a CI, in the first place. Herein, lies the crux of the issue. :) –  cardinal Apr 14 '12 at 22:38
can you provide a direct link/page reference to that example? I searched the chapter but I could not identify the correct example. –  Ronald Apr 15 '12 at 0:12
@Ronald: It's the first one on the first page of Chapter 2. A direct link would be a welcome addition. –  cardinal Apr 15 '12 at 0:15
Link as requested. Ah yes. Within this example, it seems clear: if we do an experiment, there is a 75% chance that the resulting Confidence Interval will contain the mean. Once we've done the experiment and we know how it played out, that probability may be different, depending on the distribution of the resulting sample. –  Ronald Apr 15 '12 at 0:28

Say that the CI you calculated from the particular set of data you have is one of the 5% of possible CIs that does not contain the mean. How close is it to being the 95% credible interval that you would like to imagine it to be? (That is, how close is it to containing the mean with 95% probability?) You have no assurance that it's close at all. In fact, your CI may not overlap with even a single one of the 95% of 95% CIs which do actually contain the mean. Not to mention that it doesn't contain the mean itself, which also suggests it's not a 95% credible interval.

Maybe you want to ignore this and optimistically assume that your CI is one of the 95% that does contain the mean. OK, what do we know about your CI, given that it's in the 95%? That it contains the mean, but perhaps only way out at the extreme, excluding everything else on the other side of the mean. Not likely to contain 95% of the distribution.

Either way, there's no guarantee, perhaps not even a reasonable hope that your 95% CI is a 95% credible interval.

-
I'm curious about the first paragraph. Perhaps I am misreading it, but the argument seems a little at odds with the fact that there are multiple examples in which CIs and credible intervals coincide for all possible sets of observations. What have I missed? –  cardinal Apr 15 '12 at 2:10
@cardinal: I may be wrong. I was talking the general case, but my guess would be that in the case where CI and credible interval are the same, there are other restrictions such as normality that keep the CI's from being too far afield. –  Wayne Apr 15 '12 at 2:28
My focus was drawn most strongly to the last sentence in the paragraph; the example of coincident intervals was meant to highlight a point. You might consider whether or not you truly believe that sentence or not. :) –  cardinal Apr 15 '12 at 2:33
Do you mean that a 95% CI does not imply that 5% do not include the mean? I should say "by definition, is need not even contain the mean itself"? Or am I missing even more? –  Wayne Apr 15 '12 at 2:40
Wayne, how does the fact that a particular interval not contain the mean preclude it from being a valid credible interval? Am I misreading this remark? –  cardinal Apr 15 '12 at 3:10

There are some interesting answers here, but I thought I'd add a little hands-on demonstration using R. We recently used this code in a stats course to highlight how confidence intervals work. Here's what the code does:

1 - It samples from a known distribution (n=1000)

2 - It calculates the 95% CI for the mean of each sample

3 - It asks whether or not each sample's CI includes the true mean.

4 - It reports in the console the fraction of CIs that included the true mean.

I just ran the script a bunch of times and it's actually not too uncommon to find that less than 94% of the CIs contained the true mean. At least to me, this helps dispel the idea that a confidence interval has a 95% probability of containing the true parameter.

#   In the following code, we simulate the process of
#   sampling from a distribution and calculating
#   a confidence interval for the mean of that
#   distribution.  How often do the confidence
#   intervals actually include the mean? Let's see!
#
#   You can change the number of replicates in the
#   first line to change the number of times the
#   loop is run (and the number of confidence intervals
#   that you simulate).
#
#   The results from each simulation are saved to a
#   data frame.  In the data frame, each row represents
#   the results from one simulation or replicate of the
#   loop.  There are three columns in the data frame,
#   one which lists the lower confidence limits, one with
#   the higher confidence limits, and a third column, which
#   I called "Valid" which is either TRUE or FALSE
#   depending on whether or not that simulated confidence
#   interval includes the true mean of the distribution.
#
#   To see the results of the simulation, run the whole
#   code at once, from "start" to "finish" and look in the
#   console to find the answer to the question.

#   "start"

replicates <- 1000

conf.int.low <- rep(NA, replicates)
conf.int.high <- rep(NA, replicates)
conf.int.check <- rep(NA, replicates)

for (i in 1:replicates) {

n <- 10
mu <- 70
variance <- 25
sigma <- sqrt(variance)
sample <- rnorm(n, mu, sigma)
se.mean <- sigma/sqrt(n)
sample.avg <- mean(sample)
prob <- 0.95
alpha <- 1-prob
q.alpha <- qnorm(1-alpha/2)
low.95 <- sample.avg - q.alpha*se.mean
high.95 <- sample.avg + q.alpha*se.mean

conf.int.low[i] <- low.95
conf.int.high[i] <- high.95
conf.int.check[i] <- low.95 < mu & mu < high.95
}

# Collect the intervals in a data frame
ci.dataframe <- data.frame(
LowerCI=conf.int.low,
UpperCI=conf.int.high,
Valid=conf.int.check
)

# Take a peak at the top of the data frame

# What fraction of the intervals included the true mean?
ci.fraction <- length(which(conf.int.check, useNames=TRUE))/replicates
ci.fraction

#   "finish"


Hope this helps!

-
Apologies for the criticism, but I have had to (temporarily) downvote this answer. I believe it is misunderstanding the meaning of a confidence interval and I sincerely hope this was not the argument used in your class. The simulations reduce to a (quite elaborate) binomial sampling experiment. –  cardinal Apr 15 '12 at 3:05
@cardinal Well...he's just using the long-run interpretation of frequentist statistics. Sample from the population many times, calculate the C.I. that many times and you find that the true mean is contained in the C.I. 95% of the time (for $1-\alpha=0.95$). At least that was pretty clear to me. –  Néstor Apr 15 '12 at 3:28
@Nesp: Your English is nearly flawless; I have no doubt you are understanding. I just might be focusing on the wrong aspect of the answer. I'm usually very hesitant to edit other people's answers, especially if it could substantially change the meaning or intent. As mentioned earlier, the paragraph in question seems to conflate sampling variation with some deeper interpretation of CIs. Saludos y gracias por el diálogo. –  cardinal Apr 15 '12 at 3:44
"Less than 94%" in a sample of 1000 CIs is surely not significant evidence against the idea that 95% of CIs contain the mean. In fact, I would expect 95% of CIs to indeed contain the mean, in this case. –  Ronald Apr 15 '12 at 10:53
@Ronald: Yes, this was exactly my point with the comments, but you have said it much more simply and concisely. Thanks. As stated in one of the comments, one will see 940 successes or less about 8.7% of the time and that is true of any exactly 95% CI that one constructs over the course of 1000 experiments. :) –  cardinal Apr 15 '12 at 11:37