Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

Question

It seems that through various related questions here, there is consensus that the "95%" part of what we call a "95% confidence interval" refers to the fact that if we were to exactly replicate our sampling and CI-computation procedures many times, 95% of thusly computed CIs would contain the population mean. It also seems to be the consensus that this definition does not permit one to conclude from a single 95%CI that there is a 95% chance that the mean falls somewhere within the CI. However, I don't understand how the former doesn't imply the latter insofar as, having imagined many CIs 95% of which contain the population mean, shouldn't our uncertainty (with regards to whether our actually-computed CI contains the population mean or not) force us to use the base-rate of the imagined cases (95%) as our estimate of the probability that our actual case contains the CI?

I've seen posts argue along the lines of "the actually-computed CI either contains the population mean or it doesn't, so its probability is either 1 or 0", but this seems to imply a strange definition of probability that is dependent on unknown states (i.e. a friend flips fair coin, hides the result, and I am disallowed from saying there is a 50% chance that it's heads).

Surely I'm wrong, but I don't see where my logic has gone awry...

Answer 1

Part of the issue is that the frequentist definition of a probability doesn't allow a nontrivial probability to be applied to the outcome of a particular experiment, but only to some fictitious population of experiments from which this particular experiment can be considered a sample. The definition of a CI is confusing as it is a statement about this (usually) fictitious population of experiments, rather than about the particular data collected in the instance at hand. So part of the issue is one of the definition of a probability: The idea of the true value lying within a particular interval with probability 95% is inconsistent with a frequentist framework.

Another aspect of the issue is that the calculation of the frequentist confidence doesn't use all of the information contained in the particular sample relevant to bounding the true value of the statistic. My question "Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals" discusses a paper by Edwin Jaynes which has some really good examples that really highlight the difference between confidence intervals and credible intervals. One that is particularly relevant to this discussion is Example 5, which discusses the difference between a credible and a confidence interval for estimating the parameter of a truncated exponential distribution (for a problem in industrial quality control). In the example he gives, there is enough information in the sample to be certain that the true value of the parameter lies nowhere in a properly constructed 90% confidence interval!

This may seem shocking to some, but the reason for this result is that confidence intervals and credible intervals are answers to two different questions, from two different interpretations of probability.

The confidence interval is the answer to the request: "Give me an interval that will bracket the true value of the parameter in $100p$% of the instances of an experiment that is repeated a large number of times." The credible interval is an answer to the request: "Give me an interval that brackets the true value with probability $p$ given the particular sample I've actually observed." To be able to answer the latter request, we must first adopt either (a) a new concept of the data generating process or (b) a different concept of the definition of probability itself.

The main reason that any particular 95% confidence interval does not imply a 95% chance of containing the mean is because the confidence interval is an answer to a different question, so it is only the right answer when the answer to the two questions happens to have the same numerical solution.

In short, credible and confidence intervals answer different questions from different perspectives; both are useful, but you need to choose the right interval for the question you actually want to ask. If you want an interval that admits an interpretation of a 95% (posterior) probability of containing the true value, then choose a credible interval (and, with it, the attendant conceptualization of probability), not a confidence interval. The thing you ought not to do is to adopt a different definition of probability in the interpretation than that used in the analysis.

Thanks to @cardinal for his refinements!

Here is a concrete example, from David MaKay's excellent book "Information Theory, Inference and Learning Algorithms" (page 464):

Let the parameter of interest be $\theta$ and the data $D$, a pair of points $x_1$ and $x_2$ drawn independently from the following distribution:

$p(x|\theta) = \left\{\begin{array}{cl} 1/2 & x = \theta,\\1/2 & x = \theta + 1, \\ 0 & \mathrm{otherwise}\end{array}\right.$

If $\theta$ is $39$, then we would expect to see the datasets $(39,39)$, $(39,40)$, $(40,39)$ and $(40,40)$ all with equal probability $1/4$. Consider the confidence interval

$[\theta_\mathrm{min}(D),\theta_\mathrm{max}(D)] = [\mathrm{min}(x_1,x_2), \mathrm{max}(x_1,x_2)]$.

Clearly this is a valid 75% confidence interval because if you re-sampled the data, $D = (x_1,x_2)$, many times then the confidence interval constructed in this way would contain the true value 75% of the time.

Now consider the data $D = (29,29)$. In this case the frequentist 75% confidence interval would be $[29, 29]$. However, assuming the model of the generating process is correct, $\theta$ could be 28 or 29 in this case, and we have no reason to suppose that 29 is more likely than 28, so the posterior probability is $p(\theta=28|D) = p(\theta=29|D) = 1/2$. So in this case the frequentist confidence interval is clearly not a 75% credible interval as there is only a 50% probability that it contains the true value of $\theta$, given what we can infer about $\theta$ from this particular sample.

Yes, this is a contrived example, but if confidence intervals and credible intervals were not different, then they would still be identical in contrived examples.

Note the key difference is that the confidence interval is a statement about what would happen if you repeated the experiment many times, the credible interval is a statement about what can be inferred from this particular sample.

Answer 2

In frequentist statistics probabilities are about events in the long run. They just don't apply to a single event after it's done. And the running of an experiment and calculation of the CI is just such an event.

You wanted to compare it to the probability of a hidden coin being heads but you can't. You can relate it to something very close. If your game had a rule where you must state after the flip "heads" then the probability you'll be correct in the long run is 50% and that is analogous.

When you run your experiment and collect your data then you've got something similar to the actual flip of the coin. The process of the experiment is like the process of the coin flipping in that it generates $\mu$ or it doesn't just like the coin is heads or it's not. Once you flip the coin, whether you see it or not, there is no probability that it's heads, it's either heads or it's not. Now suppose you call heads. That's what calculating the CI is. Because you can't ever reveal the coin (your analogy to an experiment would vanish). Either you're right or you're wrong, that's it. Does it's current state have any relation to the probability of it coming up heads on the next flip, or that I could have predicted what it is? No. The process by which the head is produced has a 0.5 probability of producing them but it does not mean that a head that already exists has a 0.5 probability of being. Once you calculate your CI there is no probability that it captures $\mu$, it either does or it doesn't—you've already flipped the coin.

OK, I think I've tortured that enough. The critical point is really that your analogy is misguided. You can never reveal the coin; you can only call heads or tails based on assumptions about coins (experiments). You might want to make a bet afterwards on your heads or tails being correct but you can't ever collect on it. Also, it's a critical component of the CI procedure that you're stating the value of import is in the interval. If you don't then you don't have a CI (or at least not one at the stated %).

Probably the thing that makes the CI confusing is it's name. It's a range of values that either do or don't contain $\mu$. We think they contain $\mu$ but the probability of that isn't the same as the process that went into developing it. The 95% part of the 95% CI name is just about the process. You can calculate a range that you believe afterwards contains $\mu$ at some probability level but that's a different calculation and not a CI.

It's better to think of the name 95% CI as a designation of a kind of measurement of a range of values that you think plausibly contain $\mu$ and separate the 95% from that plausibility. We could call it the Jennifer CI while the 99% CI is the Wendy CI. That might actually be better. Then, afterwards we can say that we believe $\mu$ is likely to be in the range of values and no one would get stuck saying that there is a Wendy probability that we've captured $\mu$. If you'd like a different designation I think you should probably feel free to get rid of the "confidence" part of CI as well (but it is an interval).

Answer 3

Formal, explicit ideas about arguments, inference and logic originated, within the Western tradition, with Aristotle. Aristotle wrote about these topics in several different works (including one called the Topics ;-) ). However, the most basic single principle is The Law of Non-contradiction, which can be found in various places, including Metaphysics book IV, chapters 3 & 4. A typical formulation is: " ...it is impossible for anything at the same time to be and not to be [in the same sense]" (1006 a 1). Its importance is stated slightly earlier, " ...this is naturally the starting-point even for all the other axioms" (1005 b 30). Forgive me for waxing philosophical, but this question by its nature has philosophical content that cannot simply be pushed aside for convenience.

Consider this thought-experiment: Alex flips a coin, catches it and turns it over onto his forearm with his hand covering the side facing up. Bob was standing in just the right position; he briefly saw the coin in Alex's hand, and thus can deduce which side is facing up now. However, Carlos did not see the coin--he wasn't in the right spot. At this point, Alex asks them what the probability is that the coin shows heads. Carlos suggests that the probability is .5, as that is the long-run frequency of heads. Bob disagrees, he confidently asserts that the probability is nothing else but exactly 0.

Now, who is right? It is possible, of course, that Bob mis-saw and is incorrect (let us assume that he did not mis-see). Nonetheless, you cannot hold that both are right and hold to the law of non-contradiction. (I suppose that if you don't believe in the law of non-contradiction, you could think they're both right, or some other such formulation.) Now imagine a similar case, but without Bob present, could Carlos' suggestion be more right (eh?) without Bob around, since no one saw the coin? The application of the law of non-contradiction is not quite as clear in this case, but I think it is obvious that the parts of the situation that seem to be important are held constant from the former to the latter. There have been many attempts to define probability, and in the future there may still yet be many more, but a definition of probability as a function of who happens to be standing around and where they happen to be positioned has little appeal. At any rate (guessing by your use of the phrase "confidence interval"), we are working within the Frequentist approach, and therein whether anyone knows the true state of the coin is irrelevant. It is not a random variable--it is a realized value and either it shows heads, or it shows tails.

As @John notes, the state of a coin may not at first seem similar to the question of whether a confidence interval covers the true mean. However, instead of a coin, we can understand this abstractly as a realized value drawn from a Bernoulli distribution with parameter $p$. In the coin situation, $p=.5$, whereas for a 95% CI, $p=.95$. What's important to realize in making the connection is that the important part of the metaphor isn't the $p$ that governs the situation, but rather that the flipped coin or the calculated CI is a realized value, not a random variable.

It is important for me to note at this point that all of this is the case within a Frequentist conception of probability. The Bayesian perspective does not violate the law of non-contradiction, it simply starts from different metaphysical assumptions about the nature of reality (more specifically about probability). Others on CV are much better versed in the Bayesian perspective than I am, and perhaps they may explain why the assumptions behind your question do not apply within the Bayesian approach, and that in fact, there may well be a 95% probability of the mean lying within a 95% credible interval, under certain conditions including (among others) that the prior used was accurate (see the comment by @DikranMarsupial below). However, I think all would agree, that once you state you are working within the Frequentist approach, it cannot be the case that the probability of the true mean lying within any particular 95% CI is .95.

Answer 4

Why does a 95% CI not imply a 95% chance of containing the mean?

There are many issues to be clarified in this question and in the majority of the given responses. I shall confine myself only to two of them.

a. What is a population mean? Does exist a true population mean?

The concept of population mean is model-dependent. As all models are wrong, but some are useful, this population mean is a fiction that is defined just to provide useful interpretations. The fiction begins with a probability model.

The probability model is defined by the triplet $$(\mathcal{X}, \mathcal{F}, P),$$ where $\mathcal{X}$ is the sample space (a non-empty set), $\mathcal{F}$ is a family of subsets of $\mathcal{X}$ and $P$ is a well-defined probability measure defined over $\mathcal{F}$ (it governs the data behavior). Without loss of generality, consider only the discrete case. The population mean is defined by $$ \mu = \sum_{x \in \mathcal{X}} xP(X=x), $$ that is, it represents the central tendency under $P$ and it can also be interpreted as the center of mass of all points in $\mathcal{X}$, where the weight of each $x \in \mathcal{X}$ is given by $P(X=x)$.

In the probability theory, the measure $P$ is considered known, therefore the population mean is accessible through the above simple operation. However, in practice, the probability $P$ is hardly known. Without a probability $P$, one cannot describe the probabilistic behavior of the data. As we cannot set a precise probability $P$ to explain the data behavior, we set a family $\mathcal{M}$ containing probability measures that possibly govern (or explain) the data behavior. Then, the classical statistical model emerges $$(\mathcal{X}, \mathcal{F}, \mathcal{M}).$$ The above model is said to be a parametric model if there exists $\Theta \subseteq \mathbb{R}^p$ with $p< \infty$ such that $\mathcal{M} \equiv \{P_\theta: \ \theta \in \Theta\}$. Let us consider just the parametric model in this post.

Notice that, for each probability measure $P_\theta \in \mathcal{M}$, there is a respective mean definition $$\mu_\theta = \sum_{x \in \mathcal{X}} x P_\theta(X=x).$$ That is, there is a family of population means $\{\mu_\theta: \ \theta \in \Theta\}$ that depends tightly on the definition of $\mathcal{M}$. The family $\mathcal{M}$ is defined by limited humans and therefore it may not contain the true probability measure that governs the data behavior. Actually, the chosen family will hardly contain the true measure, moreover this true measure may not even exist. As the concept of a population mean depends on the probability measures in $\mathcal{M}$, the population mean is model-dependent.

The Bayesian approach considers a prior probability over the subsets of $\mathcal{M}$ (or, equivalently, $\Theta$), but in this post I will concentrated only on the classical version.

b. What is the definition and the purpose of a confidence interval?

As aforementioned, the population mean is model-dependent and provides useful interpretations. However, we have a family of population means, because the statistical model is defined by a family of probability measures (each probability measure generates a population mean). Therefore, based on an experiment, inferential procedures should be employed in order to estimate a small set (interval) containing good candidates of population means. One well-known procedure is the ($1-\alpha$) confidence region, which is defined by a set $C_\alpha$ such that, for all $\theta \in \Theta$, $$ P_\theta(C_\alpha(X) \ni \mu_\theta) \geq 1-\alpha \ \ \ \mbox{and} \ \ \ \inf_{\theta\in \Theta} P_\theta(C_\alpha(X) \ni \mu_\theta) = 1-\alpha, $$ where $P_\theta(C_\alpha(X) = \varnothing) = 0$ (see Schervish, 1995). This is a very general definition and encompasses virtually any type of confidence intervals. Here, $P_\theta(C_\alpha(X) \ni \mu_\theta)$ is the probability that $C_\alpha(X)$ contains $\mu_\theta$ under the measure $P_\theta$. This probability should be always greater than (or equal to) $1-\alpha$, the equality occurs at the worst case.

Remark: The readers should notice that it is not necessary to make assumptions on the state of reality, the confidence region is defined for a well-defined statistical model without making reference to any "true" mean. Even if the "true" probability measure does not exist or it is not in $\mathcal{M}$, the confidence region definition will work, since the assumptions are about statistical modelling rather than the states of reality.

On the one hand, before observing the data, $C_\alpha(X)$ is a random set (or random interval) and the probability that "$C_\alpha(X)$ contains the mean $\mu_\theta$" is, at least, $(1-\alpha)$ for all $\theta \in \Theta$. This is a very desirable feature for the frequentist paradigm.

On the other hand, after observing the data $x$, $C_\alpha(x)$ is just a fixed set and the probability that "$C_\alpha(x)$ contains the mean $\mu_\theta$" should be in {0,1} for all $\theta \in \Theta$.

That is, after observing the data $x$, we cannot employ the probabilistic reasoning anymore. As far as I know, there is no theory to treat confidence sets for an observed sample (I am working on it and I am getting some nice results). For a while, the frequentist must believe that the observed set (or interval) $C_\alpha(x)$ is one of the $(1-\alpha)100\%$ sets that contains $\mu_\theta$ for all $\theta\in \Theta$.

PS: I invite any comments, reviews, critiques, or even objections to my post. Let's discuss it in depth. As I am not a native English speaker, my post surely contains typos and grammar mistakes.

Reference:

Schervish, M. (1995), Theory of Statistics, Second ed, Springer.

Answer 5

I'm surprised that no one has brought up Berger's example of an essentially useless 75% confidence interval described in the second chapter of "The Likelihood Principle". The details can be found in the original text (which is available for free on Project Euclid): what is essential about the example is that it describes, unambiguously, a situation in which you know with absolute certainty the value of an ostensibly unknown parameter after observing data, but you would assert that you have only 75% confidence that your interval contains the true value. Working through the details of that example was what enabled me to understand the entire logic of constructing confidence intervals.

Edit: The Project Euclid link appears to be broken as of 2022-01-21. The monograph can be found e.g. here or here.

Answer 6

I don't know whether this should be asked as a new question but it is addressing the very same question asked above by proposing a thought experiment.

Firstly, I'm going to assume that if I select a playing card at random from a standard deck, the probability that I've selected a club (without looking at it) is 13 / 52 = 25%.

And secondly, it's been stated many times that a 95% confidence interval should be interpreted in terms of repeating an experiment multiple times and the calculated interval will contain the true mean 95% of the time – I think this was demonstated reasonably convincingly by James Waters simulation. Most people seem to accept this interpretation of a 95% CI.

Now, for the thought experiment. Let's assume that we have a normally distributed variable in a large population - maybe heights of adult males or females. I have a willing and tireless assistant whom I task with performing multiple sampling processes of a given sample size from the population and calculating the sample mean and 95% confidence interval for each sample. My assistant is very keen and manages to measure all possible samples from the population. Then, for each sample, my assistant either records the resulting confidence interval as green (if the CI contains the true mean) or red (if the CI doesn't contain the true mean). Unfortunately, my assistant will not show me the results of his experiments. I need to get some information about the heights of adults in the population but I only have time, resources and patience to do the experiment once. I make a single random sample (of the same sample size used by my assistant) and calculate the confidence interval (using the same equation).

I have no way of seeing my assistant's results. So, what is the probability that the random sample I have selected will yield a green CI (i.e. the interval contains the true mean)?

In my mind, this is the same as the deck of cards situation outlined previously and can be interpreted that is a 95% probability that the calculated interval contains the true mean (i.e. is green). And yet, the concensus seems to be that a 95% confidence interval can NOT be interpreted as there being a 95% probability that the interval contains the true mean. Why (and where) does my reasoning in the above thought experiment fall apart?

Answer 7

For practical purposes, you're no more wrong to bet that your 95% CI included the true mean at 95:5 odds, than you are to bet on your friend's coin flip at 50:50 odds.

If your friend already flipped the coin, and you think there's a 50% probability of it being heads, then you're just using a different definition of the word probability. As others have said, for frequentists you can't assign a probability to an event having occurred, but rather you can describe the probability of an event occurring in the future using a given process.

From another blog: The frequentist will say: "A particular event cannot have a probability. The coin shows either head or tails, and unless you show it, I simply can't say what is the fact. Only if you would repeat the toss many, many times, any if you vary the initial conditions of the tosses strongly enough, I'd expect that the relative frequency of heads in all thes many tosses will approach 0.5". http://www.researchgate.net/post/What_is_the_difference_between_frequentist_and_bayesian_probability

Answer 8

In this answer to a different question, Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals, I explained a difference between confidence intervals and credible intervals. Both intervals can be constructed such that they will contain a certain fraction of the times the true parameter. However there is a difference in the conditional dependence on the observation and the conditional dependence on the true parameter values.

• An $\alpha \%$-confidence interval will contain the parameter a fraction $\alpha \%$ of the time, independent from the true parameter. But the confidence interval will not contain the parameter a fraction $\alpha \%$ of the time, independent from the observation value.

This contrasts with

• An $\alpha \%$-credible interval will contain the parameter a fraction $\alpha \%$ of the time, independent from the observation value. But the credible interval will not contain the parameter a fraction $\alpha \%$ of the time, independent from the true parameter.

See also the image accompanying that answer:

Answer 9

The misinterpretation of confidence intervals is related to what Blitzstein and Hwang (in their probability textbook) call "sympathetic magic".

Sympathetic magic is an anthropology term for a practice by tribal folk, who believed that manipulating the representation of something would affect the actual thing. E.g. they would stick pins in the voodoo doll of a disliked person, and claim that this would hurt the actual person. Though this seems laughable, even educated people fall for it: psychologist Paul Rozin and others have demonstrated that Ivy League students are susceptible to sympathetic magic. Confidence intervals are somewhat like this: here, we misinterpret the realization of something to be indicative of the true thing. More specifically this is the "Fallacy of Reification", but I feel like voodoo dolls is a clearer analogy.

Ask yourself: what exactly is a confidence interval of confidence level $100(1-\alpha)\%$ ? Is it a pair of numbers that we calculate from a realization of the sample? e.g. for a realization $(x_1=0.89, x_2=-0.17, ..., x_n=1.23,)$, is a confidence interval the pair $(\hat{\theta}_{L}=0.53, \hat{\theta}_{H}=0.57)$? The answer is no; although we do informally say "the 95% confidence interval is (0.53, 0.57)", this is better called a realization of a confidence interval or confidence interval estimate; it is not itself a confidence interval.

To be specific:

A confidence interval is a pair of random variables $(\hat{\theta}_{L}, \hat{\theta}_{H})$ which is output from some procedure $h_{\alpha}(\cdot)$ that acts on the sample $X_1, ..., X_n$, i.e. $(\hat{\theta}_{L}, \hat{\theta}_{H}) = h_{\alpha}(X_1, ..., X_n)$.

Compare this to the definition of an estimator:

An estimator is a random variable $\hat{\theta}_{n}$ which is computed from an estimation procedure $h(\cdot)$ that acts on the sample $X_1, ..., X_n$, i.e. $\hat{\theta}_{n} = h(X_1, ..., X_n)$.

If we have a realization of the sample $(x_1=0.89, x_2=-0.17, ..., x_n=1.23,)$, it would be incorrect to call $0.55$ the estimator; the estimator is a random variable, whereas $0.55$ is a number. It would be better to call $0.55$ a realization of the estimator (as @GrahamBornholt notes, estimate is more appropriate, but I will use "realization" to make the difference clear).

The above point is really important; there are actually three quantities we are dealing with:

1. The confidence interval estimation procedure $h_{\alpha}(\cdot)$, which is a function that computes something from the sample $X_1, ..., X_n$.
2. The confidence interval $(\hat{\theta}_{L}, \hat{\theta}_{H})$, which is a pair of random variables.
3. The realization of a confidence interval, which is a pair of real values like $(0.53, 0.57)$.

When learning statistics, we don't care about the CI realization, because it is specific to both our problem, and the realization of the sample. However, we care a lot about the confidence interval estimation procedure i.e. $h_{\alpha}(\cdot)$. The only requirement of $h_{\alpha}(\cdot)$ to be called a valid CI procedure i.e. which produces a "$100(1-\alpha)\%$ confidence interval" from a sample, is the following:

If you realize an infinite number of samples $(x_1^{(1)}, ..., x_n^{(1)}), (x_1^{(2)}, ..., x_n^{(2)}), ...$ and compute the realized CI for each, i.e. $(\hat{\theta}_{L}^{(1)}, \hat{\theta}_{H}^{(1)}), { } (\hat{\theta}_{L}^{(2)}, \hat{\theta}_{H}^{(2)}), ...$, then at least $100(1-\alpha)\%$ of these realized CIs will contain the true population parameter $\theta$.

That's it. That's the criterion for calling something a CI estimation procedure. Any function which does this, is by definition a CI procedure, and produces CIs. This includes degenerate functions e.g. one which always outputs $(-\infty, \infty)$ as the CI; it contains the true $\theta$ 100% of the time, and $100(1-\alpha)\% \le 100\%$, so its valid CI procedure.

So, there's no "one true CI procedure"; there are many valid functions $h_{\alpha}(\cdot)$ for an estimator, and some are better than others. Obviously $(-\infty, \infty)$ was a useless CI, but it illustrates one desirable property of CIs estimation procedures: we want a procedure $h_{\alpha}(\cdot)$ which outputs a $100(1-\alpha)\%$ CI which is as "tight" as possible, i.e. where $|\hat{\theta}_{H} - \hat{\theta}_{L}|$ is as small as possible, over all possible samples.

Given all this setup, let's revisit the original question: why doesn't a 95% confidence interval imply that there is a 95% probability that the true $\theta$ is in the confidence interval?

The confusion here lies in what we are calling a "confidence interval". Obviously, it is nonsensical to say that $\theta$ lies in the estimation procedure (a function), so we'll consider the other two possibilities:

• Are we talking about a realized confidence interval, e.g. $(0.53, 0.57)$ ?
• Are we talking about the formal definition of a confidence interval i.e. the pair of random variables $(\hat{\theta}_{L}, \hat{\theta}_{H})$ ?

If we are talking about the pair of r.v.s, then the statement is correct. But if we are abusing the phase "confidence interval" to refer to a particular realization, then the statement is wrong.

The explanation proceeds as follows:

• Recall, our random experiment is drawing a sample $X_1, ..., X_n$. Every time we run this experiment, we get a realized sample of $n$ numbers. Since our sampling procedure is assumed to have a randomization step, there is uncertainty in what we will draw, and hence $X_1, ..., X_n$ are random variables. The sample space $\Omega$ is the set of all possible samples. The sample is the only random part of this whole setup.
• The basic assumption in frequentist statistics is that our population is fixed. Thus, $\theta$ is a constant, e.g. the mean of our population. It's an unknown variable, but not a random variable, because its value is not calculated from the random sample. Also, it stays the same no matter how many samples we draw. Think of it as a number written in thousand-foot-tall letters on some distant planet; it never erodes, but you will never know it.
• Consider the event $(\hat{\theta}_{L} \le \theta \le \hat{\theta}_{H})$ i.e. $\theta \in h_{\textrm{alpha}}(X_1, ..., X_n)$. This is an event defined using random variables, similar to simpler events like $(Y<5)$. It has a probability, namely $p(\hat{\theta}_{L} \le \theta \le \hat{\theta}_{H})$. By definition of the CI procedure, this probability will be at least $100(1-\alpha)\%$. So long as we are talking about random variables, we are good.
• Consider the realization of a CI, (0.53, 0.57). What's the probability that $0.53 \le \theta \le 0.57$? The answer is that it's not a probability at all, it's a true-or-false statement about real numbers, similar to statements like "$3 \le 5$" (true) or "$6 \le 5$" (false). Remember: by assumption, our population is fixed, so $\theta$ is a number, not a random variable. There's no uncertainty associated with the statement $0.53 \le \theta \le 0.57$, it's just a fact about numbers. You can't even make a Bernoulli out of it, because that Bernoulli would be degenerate, with p=0 (false) or p=1 (true).

If the final point leaves something to be desired, because you want $\theta$ to be a varying quantity, you are in luck; that's exactly what Bayesian statistical intervals attempt to address.

Answer 10

While there has been extensive discussion in the numerous great answers, I want to add a more simple perspective. (although it has been alluded in other answers - but not explicitly.) For some parameter $\theta$, and given a sample $(X_1,X_2,\cdots,X_n)$, a $100p\%$ confidence interval is a probability statement of the form

$$P\left(g(X_1,X_2,\cdots,X_n)<\theta<f(X_1,X_2,\cdots,X_n)\right)=p$$

If we consider $\theta$ to be a constant, then the above statement is about the random variables $g(X_1,X_2,\cdots,X_n)$ and $f(X_1,X_2,\cdots,X_n)$, or more accurately, it is about the random interval $\left(g(X_1,X_2,\cdots,X_n),f(X_1,X_2,\cdots,X_n)\right)$.

So instead of giving any information about the probability of the parameter being contained in the interval, it is giving information about the probability of the interval containing the parameter - as the interval is made from random variables.

Answer 11

It all depends on whether you are looking at the probability conditional or unconditional on the data. Suppose you have an unknown parameter $\theta \in \Theta$ and you make a confidence interval for this parameter using sample data $\mathbf{x}$. Let $\text{CI}_\theta(\mathbf{X},1-\alpha)$ denote the (random) confidence interval at confidence level $1-\alpha$ and with (random) data $\mathbf{X}$. An exact confidence interval satisfies the following conditional probability condition:

$$\mathbb{P}(\theta \in \text{CI}_\theta(\mathbf{X},1-\alpha) | \theta) = 1-\alpha \quad \quad \quad \quad \quad \text{for all } \theta \in \Theta.$$

If we are willing to ascribe a probability distribution to $\theta$ (e.g., as in Bayesian analysis) this also implies the marginal probability that:

$$\mathbb{P}(\theta \in \text{CI}_\theta(\mathbf{X},1-\alpha)) = 1-\alpha.$$

However, it is not generally true that:

$$\mathbb{P}(\theta \in \text{CI}_\theta(\mathbf{X},1-\alpha) | \mathbf{X} = \mathbf{x}) = 1-\alpha.$$

---

As you can see from the above, if we are looking at the probability unconditional on the data (and either conditional or unconditional on the parameter) then we can say that the probability of the unknown quantity falling into the confidence interval is equal to the confidence level. However, if we are looking at the probability conditional on the data we cannot say that the probability of the unknown quantity falling into the confidence interval is equal to the confidence level.

Typically, we frame this by saying that the confidence interval procedure/method (considered prior to substitution of the data) will cover the true parameter with probability equal to the confidence level, but once we have an actual confidence interval (i.e., after substituting the observed data and conditioning our probability statements on the data) this probability statement no longer holds. This is the reason we refer to having 95% "confidence" rather than 95% probability for the parameter being in the interval.

Answer 12

Say that the CI you calculated from the particular set of data you have is one of the 5% of possible CIs that does not contain the mean. How close is it to being the 95% credible interval that you would like to imagine it to be? (That is, how close is it to containing the mean with 95% probability?) You have no assurance that it's close at all. In fact, your CI may not overlap with even a single one of the 95% of 95% CIs which do actually contain the mean. Not to mention that it doesn't contain the mean itself, which also suggests it's not a 95% credible interval.

Maybe you want to ignore this and optimistically assume that your CI is one of the 95% that does contain the mean. OK, what do we know about your CI, given that it's in the 95%? That it contains the mean, but perhaps only way out at the extreme, excluding everything else on the other side of the mean. Not likely to contain 95% of the distribution.

Either way, there's no guarantee, perhaps not even a reasonable hope that your 95% CI is a 95% credible interval.

Answer 13

First, let's give a definition of the confidence interval, or, in spaces of dimension greater than one, the confidence region. The definition is a concise version of that given by Jerzy Neyman in his 1937 paper to the Royal Society.

Let the parameter be $\mathfrak{p}$ and the statistic be $\mathfrak{s}$. Each possible parameter value $p$ is associated with an acceptance region $\mathcal{A}(p,\alpha)$ for which $\mathrm{prob}(\mathfrak{s} \in \mathcal{A}(p,\alpha) | \mathfrak{p} = p, \mathcal{I}) = \alpha$, with $\alpha$ being the confidence coefficient, or confidence level (typically 0.95), and $\mathcal{I}$ being the background information which we have to define our probabilities. The confidence region for $\mathfrak{p}$, given $\mathfrak{s} = s$, is then $\mathcal{C}(s,\alpha) = \{p | s \in \mathcal{A}(p,\alpha)\}$.

In other words, the parameter values which form the confidence region are just those whose corresponding $\alpha$-probability region of the sample space contains the statistic.

Now consider that for any possible parameter value $p$:

\begin{align} \int{[p \in \mathcal{C}(s,\alpha)]\:\mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I})}\:ds &= \int{[s \in \mathcal{A}(p,\alpha)]\:\mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I})}\:ds \\ &= \alpha \end{align}

where the square brackets are Iverson brackets. This is the key result for a confidence interval or region. It says that the expectation of $[p \in \mathcal{C}(s,\alpha)]$, under the sampling distribution conditional on $p$, is $\alpha$. This result is guaranteed by the construction of the acceptance regions, and moreover it applies to $\mathfrak{p}$, because $\mathfrak{p}$ is a possible parameter value. However, it is not a probability statement about $\mathfrak{p}$, because expectations are not probabilities!

The probability for which that expectation is commonly mistaken is the probability, conditional on $\mathfrak{s} = s$, that the parameter lies in the confidence region:

$$ \mathrm{prob}(\mathfrak{p} \in \mathcal{C}(s,\alpha) | \mathfrak{s} = s, \mathcal{I}) = \frac{\int_{\mathcal{C}(s,\alpha)} \mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I}) \:\mathrm{prob}(\mathfrak{p} = p | \mathcal{I}) \: dp}{\int \mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I}) \:\mathrm{prob}(\mathfrak{p} = p | \mathcal{I}) \: dp} $$

This probability reduces to $\alpha$ only for certain combinations of information $\mathcal{I}$ and acceptance regions $\mathcal{A}(p,\alpha)$. For example, if the prior is uniform and the sampling distribution is symmetric in $s$ and $p$ (e.g. a Gaussian with $p$ as the mean), then:

\begin{align} \mathrm{prob}(\mathfrak{p} \in \mathcal{C}(s,\alpha) | \mathfrak{s} = s, \mathcal{I}) &= \frac{\int_{\mathcal{C}(s,\alpha)} \mathrm{prob}(\mathfrak{s} = p | \mathfrak{p} = s, \mathcal{I}) \: dp}{\int \mathrm{prob}(\mathfrak{s} = p | \mathfrak{p} = s, \mathcal{I}) \: dp} \\ &= \mathrm{prob}(\mathfrak{s} \in \mathcal{C}(s,\alpha) | \mathfrak{p} = s, \mathcal{I}) \\ &= \mathrm{prob}(s \in \mathcal{A}(\mathfrak{s},\alpha) | \mathfrak{p} = s, \mathcal{I}) \end{align}

If in addition the acceptance regions are such that $s \in \mathcal{A} (\mathfrak{s},\alpha) \iff \mathfrak{s} \in \mathcal{A}(s,\alpha)$, then:

\begin{align} \mathrm{prob}(\mathfrak{p} \in \mathcal{C}(s,\alpha) | \mathfrak{s} = s, \mathcal{I}) &= \mathrm{prob}(\mathfrak{s} \in \mathcal{A}(s,\alpha) | \mathfrak{p} = s, \mathcal{I}) \\ &= \alpha \end{align}

The textbook example of estimating a population mean with a standard confidence interval constructed about a normal statistic is a special case of the preceding assumptions. Therefore the standard 95% confidence interval does contain the mean with probability 0.95; but this correspondence does not generally hold.

Answer 14

What one should not say when using frequentist inference is, "There is 95% probability that the unknown fixed true theta is within the computed confidence interval." To the frequentist probability describes the emergent pattern over many (observable!) samples and is not a statement about a single event. However, understanding the long-run emergent pattern gives us confidence in what to expect in a single event. The key is to replace "probability" with "confidence," i.e. "I am 95% confident that the unknown fixed true theta is within the computed confidence interval."

This is analogous to knowing the bias of a coin is 0.95 in favor of heads (95% of the time the coin lands heads) and the confidence this knowledge of the long-run proportion imbues regarding the outcome of a single flip. If asked how confident you are that the coin will land heads (or has already landed heads), you would say you are 95% confident based on its long-run performance.

To the frequentist, the limiting proportion is the probability and our confidence is a result of knowing this limiting proportion. To the Bayesian, the long-run emergent pattern over many samples is not a probability. The belief of the experimenter is the probability. The Bayesian is also willing to make (belief) probability statements about an unobservable population parameter without any connection to sampling. Such statements are not verifiable statements about the actual parameter, the hypothesis, nor the experiment. These are statements about the experimenter. The frequentist is not willing to make such statements.

Here is a related thread showing the interpretation of frequentist confidence and Bayesian belief in the context of a COVID screening test. Here is a related thread comparing frequentist and Bayesian inference for a binomial proportion near 0 or 1. To the frequentist, the Bayesian posterior can be viewed as a crude approximate p-value function showing p-values and confidence intervals of all levels.

Answer 15

In his answer, Dikran Marsupial provides the following example as evidence that no confidence interval is admissible as a set of plausible parameter values consistent with the observed data:

Let the parameter of interest be $\theta$ and the data $D$, a pair of points $x_1$ and $x_2$ drawn independently from the following distribution:

$p(x|\theta) = \left\{\begin{array}{cl} 1/2 & x = \theta,\\1/2 & x = > \theta + 1, \\ 0 & \mathrm{otherwise}\end{array}\right.$

If $\theta$ is $39$, then we would expect to see the datasets $(39,39)$, $(39,40)$, $(40,39)$ and $(40,40)$ all with equal probability $1/4$.

We are then asked to consider the confidence interval

$[\theta_\mathrm{min}(D),\theta_\mathrm{max}(D)] = [\mathrm{min}(x_1,x_2), \mathrm{max}(x_1,x_2)]$

and informed this will correctly cover the unknown fixed true $\theta$ $75\%$ of the time in repeated sampling. We are also informed that for an observed data set, $D=\{29,29\}$, the posterior belief probabilities for $\theta=28$ and $\theta=29$ are $p(\theta=28|D) = p(\theta=29|D) = 1/2$ (without reference to a prior) while the $75\%$ confidence interval is $\theta\in(29)$. Dikran Marsupial claims that since the confidence level of the confidence interval is a statement about repeated experiments it does not allow one to infer the unknown fixed true $\theta$ based on a particular sample. He further claims that only Bayesian belief is capable of such inference based on a sample.

It is best to view a confidence interval as the inversion of a hypothesis test, especially when dealing with a discrete parameter space. For this example we can use the entire data set as the test statistic when calculating the p-value.

For $H_0: 27\ge\theta\ge 30$, the probability of the observed result, $D=\{29,29\}$, or something more extreme is $0$, so we can rule out these hypotheses without error. We can therefore construct the $100\%$ confidence interval $\theta \in(28,29)$. This is a direct contradiction to Dikran's claim that a confidence interval does not allow one to infer the unknown fixed true $\theta$ based on a particular sample. This interval was constructed without any prior belief.

The remaining hypotheses available for constructing a narrower confidence interval are $H: \theta=28$ and $H:\theta=29$. Under $H_0: \theta=28$, the upper-tailed probability of the observed result, $D=\{29,29\}$, or something more extreme is $0.25$. One conclusion is to "rule out" $H_0: \theta=28$ at the $0.25$ level in favor of $H_1:\theta=29$, producing the $75\%$ confidence interval $\theta \in (29)$. Likewise, under $H_0: \theta=29$ the lower-tailed probability of the observed result, $D=\{29,29\}$, or something more extreme is $0.25$. Another conclusion is to "rule out" $H_0: \theta=29$ at the $0.25$ level in favor of $H_1:\theta=28$, producing the $75\%$ confidence interval $\theta \in (28)$.

The confidence level of these intervals is not a measure of the experimenter's belief, it is a restatement of the p-value and a measure of the interval's performance over repeated experiments. This does not preclude the confidence interval as a method for performing inference on a parameter based on a particular sample.

Dikran's posterior belief probabilities and credible intervals can instead be viewed as crude approximate p-values and confidence intervals. The $100\%$ credible interval is $(28,29)$, the posterior probability "ruling out" $H_0: \theta=28$ is $0.5$, and the posterior probability "ruling out" $H_0: \theta=29$ is $0.5$.

Answer 16

There are some interesting answers here, but I thought I'd add a little hands-on demonstration using R. We recently used this code in a stats course to highlight how confidence intervals work. Here's what the code does:

1 - It samples from a known distribution (n=1000)

2 - It calculates the 95% CI for the mean of each sample

3 - It asks whether or not each sample's CI includes the true mean.

4 - It reports in the console the fraction of CIs that included the true mean.

I just ran the script a bunch of times and it's actually not too uncommon to find that less than 94% of the CIs contained the true mean. At least to me, this helps dispel the idea that a confidence interval has a 95% probability of containing the true parameter.

#   In the following code, we simulate the process of
#   sampling from a distribution and calculating
#   a confidence interval for the mean of that 
#   distribution.  How often do the confidence
#   intervals actually include the mean? Let's see!
#
#   You can change the number of replicates in the
#   first line to change the number of times the 
#   loop is run (and the number of confidence intervals
#   that you simulate).
#
#   The results from each simulation are saved to a
#   data frame.  In the data frame, each row represents
#   the results from one simulation or replicate of the 
#   loop.  There are three columns in the data frame, 
#   one which lists the lower confidence limits, one with
#   the higher confidence limits, and a third column, which
#   I called "Valid" which is either TRUE or FALSE
#   depending on whether or not that simulated confidence
#   interval includes the true mean of the distribution.
#
#   To see the results of the simulation, run the whole
#   code at once, from "start" to "finish" and look in the
#   console to find the answer to the question.    

#   "start"

replicates <- 1000

conf.int.low <- rep(NA, replicates)
conf.int.high <- rep(NA, replicates)
conf.int.check <- rep(NA, replicates)

for (i in 1:replicates) {

        n <- 10
        mu <- 70
        variance <- 25
        sigma <- sqrt(variance)
        sample <- rnorm(n, mu, sigma)
        se.mean <- sigma/sqrt(n)
        sample.avg <- mean(sample)
        prob <- 0.95
        alpha <- 1-prob
        q.alpha <- qnorm(1-alpha/2)
        low.95 <- sample.avg - q.alpha*se.mean
        high.95 <- sample.avg + q.alpha*se.mean

        conf.int.low[i] <- low.95
        conf.int.high[i] <- high.95
        conf.int.check[i] <- low.95 < mu & mu < high.95
 }    

# Collect the intervals in a data frame
ci.dataframe <- data.frame(
        LowerCI=conf.int.low,
        UpperCI=conf.int.high, 
        Valid=conf.int.check
        )

# Take a peak at the top of the data frame
head(ci.dataframe)

# What fraction of the intervals included the true mean?
ci.fraction <- length(which(conf.int.check, useNames=TRUE))/replicates
ci.fraction

    #   "finish"

Hope this helps!