3
$\begingroup$

Today, my instructor was going over some of the misconceptions about confidence interval, e.g., how it's frequently incorrectly stated that "a 95% confidence interval that there is a 95% chance that the population parameter is in the confidence interval." He further said that "95% confidence intervals mean that if you were to create $n$ sets of samples drawn from this population, and construct a separate confidence interval for each of the $n$ sets, then we would expect 95% of these confidence intervals to contain the population parameter." This seems to be what's regurgitated on this website.

So he concluded with

(1) A 95% confidence interval does not mean that there is a 95% chance that the population parameter is in this confidence interval. It's either in it, or it's not in it, there's no probability associated with that. This makes sense to me.

(2) A 95% confidence interval means that it has a 95% chance of containing the population parameter. ???? Someone brought up that (2) sounds the exact same as (1), but just worded differently (and I agree).

Doesn't (1) = (2), or is (2) somehow different from (1)?

I tried to apply a simplified analogy where we consider a set of numbers $S$ and some target number $x$. According to (1), it's saying that $x$ has a 95% chance of being in $S$. According to (2), it's saying that $S$ has a 95% chance of containing $x$.

These two literally sound the same to me.

$\endgroup$
8
  • 1
    $\begingroup$ "This seems to be what's regurgitated on this website." There's probably a word with a better connotation that conveys your point. $\endgroup$ May 14, 2021 at 2:19
  • $\begingroup$ @AryaMcCarthy I don't think so. My particular question here is if the phrasing in (1) and (2) are actually different. $\endgroup$ May 14, 2021 at 3:41
  • 2
    $\begingroup$ This is actually addressed in the answer to the question I linked: The subtlety is that a 𝑧% confidence interval for a parameter means that the endpoints of the interval (which are random variables) lie either side of the parameter with probability 𝑧% before you calculate the interval, not that the probability of the parameter lying within the interval is 𝑧% after you have calculated the interval. $\endgroup$ May 14, 2021 at 3:52
  • $\begingroup$ To Arya's point, once you condition on your data, then the game is over. The data determine the interval estimate and so there is no more randomness to contend with. $\endgroup$ May 14, 2021 at 4:23
  • 2
    $\begingroup$ The key is that the CI is a random quantity, not the parameter. $\endgroup$
    – AdamO
    May 14, 2021 at 4:54

4 Answers 4

4
$\begingroup$

Firstly, it is not entirely correct to say that you cannot make a probability statement about a confidence interval. The set function that defines a confidence interval is actually defined by an a priori probability requirement, expressed as a probability statement that conditions on the unknown parameter but not the data. Viewed as a procedure (i.e., as a function operating on data and a chosen confidence level) it is possible to make a valuable probability statement about the confidence interval, as we will see below (see here for more formal discussion of the definition).

Suppose we have an observable data vector $\mathbf{X} \equiv (X_1,...,X_n)$ from a distribution with some unknown parameter $\theta \in \Theta$. Then a confidence interval procedure for the parameter $\theta$ is a set function $\text{CI}$ that satisfies the following conditional probability requirement:$^\dagger$

$$1-\alpha = \mathbb{P}(\theta \in \text{CI}(\mathbf{X}, \alpha)|\theta) \quad \quad \quad \text{for all } \theta \in \Theta \text{ and } 0 \leqslant \alpha \leqslant 1.$$

Note that for any prior distribution over $\theta$, this also implies the weaker probability result:

$$1-\alpha = \mathbb{P}(\theta \in \text{CI}(\mathbf{X}, \alpha)) \quad \quad \quad \text{for all } 0 \leqslant \alpha \leqslant 1. \quad \quad \quad \quad $$

The probability requirement above is an a priori probability statement that treats the data (and therefore the confidence interval) as random. Now, given a set function that comports to the above conditional probability requirement, for a given value $0 \leqslant \alpha \leqslant 1$, the confidence set for the observed data $\mathbf{x}$ (with confidence level $1-\alpha$) is the fixed set $\text{CI}(\mathbf{x}, \alpha)$. This object is no longer random, so it now either contains the parameter $\theta$ or it does not.


Properly interpreting a confidence interval: The proper interpretation of the confidence interval hinges on correctly stating the conditioning information in the probability statement. As can be seen, the probability requirement for a confidence interval is conditional on the parameter, but it is not conditional on the data (i.e., it is viewed a priori, where the data are random variables yet to be observed).

Confidence intervals can be properly interpreted either by correctly stating their probability requirement, or by stating the analogous long-run behaviour of the interval for hypothetically repeated data using the "law of large numbers". Here are some examples of correct and incorrect interpretations of a 95% confidence interval:

  • (Correct): When viewed as a procedure that has not yet been applied to the data (i.e., not conditioning on observed data), there is a 95% chance that the parameter falls within the confidence interval.

  • (Correct): Regardless of the true parameter value, the prior probability that the confidence interval procedure will lead to a confidence interval containing the parameter is 95%.

  • (Correct): If the confidence interval procedure is applied repeatedly to random data from the stipulated distribution (the one used to form the confidence interval), in the long-run the parameter will be in the confidence interval 95% of the time.

  • (Incorrect): The confidence interval formed for a given set of observed data will contain the parameter with 95% probability.

The problem with the statements given by your teacher is that they are ambiguous about what probability statement is being made (and they both seem to say the same thing in a different way). When viewed as a fixed interval after the data are observed, it is true that the interval either contains the data or does not contain it (with probability zero or one). However, when viewed as a procedure without conditioning on observed data,

I would recommend you try to get into the habit of distinguishing an actual (fixed) confidence interval formed from observed data from the confidence interval procedure that maps the data to the interval. The latter is the object for which a valuable a priori probability statement can be made.

$\endgroup$
3
$\begingroup$

If, according to (2), there is a 95% chance of the interval containing the population parameter then surely you can demonstrate this to me via simulation. That would mean you could:

  • Generate a single confidence interval
  • Somehow repeatedly examine if the population parameter is in the interval
  • Since there is a 95% chance the parameter is in the interval, then some of those examinations would show that the parameter is in the interval while others show it is not. That is what probability means to a frequentist.

That seems a bit non-nonsensical because (as you say in (1)) the parameter is either inside the interval or not. There is nothing random about the interval. The 95% refers to the long term relative frequency of the interval containing the estimand upon repeated construction.

(2) is not the same as (1), though that is a common mistake. I've written more about this here.

$\endgroup$
2
  • $\begingroup$ Consider a cookie dough. The cookie dough is cut into 100 equal sized cookies randomly. Prior to cutting, you distribute chocolate chips uniformly so that when the dough is cut into 100 equal sized cookies randomly, you would expect at least 95% of these cookies to have at least 1 chocolate chip. Is this equivalent to saying that each cookie dough has a 95% chance of having at least 1 chocolate chip? I don't think so from my perspective. The analogy here is the former seems to be what a confidence interval is all about and the latter is what statement (2) is saying. $\endgroup$ May 14, 2021 at 5:01
  • $\begingroup$ can you make such a simulation for a credible interval? $\endgroup$
    – rep_ho
    May 14, 2021 at 10:17
1
$\begingroup$

I agree, (1) and (2) are the same statement. It seems that treating them as different statements is a weird convention followed by statistics teachers.

I think the intention is to emphasise that it's the interval which is random, and not the parameter being estimated. It could be argued that statement (1) makes it sound like the population parameter is random, but it's not.

I find the following example of a confidence interval useful.

In your example, each time you run the experiment, generate a random number, and take your set (what would usually be called the "confidence interval") to be the whole set $S$ with probability 95% and the empty set with probability 5%.

Then you can see that exactly 95% of these sets automatically contain $x$, so this is a confidence interval.

However, it's immediately clear that any particular set generated in this way either does or does not contain $x$, because either it's the whole of $S$ or it is empty.

Note that it's the procedure itself which is called a confidence interval, not the sets which it generates.

$\endgroup$
1
$\begingroup$

Your reasoning is correct. The instructor's statments are contradictory, because the probability that the interval contains the parameter is the same as the probability that the parameter is in the interval. In general, if $X$ and $Y$ are events such that $X$ happens if and only if $Y$ happens, then obviously $P(X) = P(Y)$. I suspect that by using the word "this" in statement 1 but not statement 2, they meant that to convey that statement 1 refers to a confidence interval which has already been generated and whose endpoints are already known, and statement 2 does not - but their choice of wording fails to convey this meaning. They should have said:

  1. prior to seeing the interval/data, you should assign probability 0.95 to the proposition that the interval contains/will contain the parameter;
  2. once you have seen the interval (or any of the data), you have new information and so this probability is out of date.

To give a concrete example, if you are going to measure the heights of 10 random men and use them to build a 95% confidence interval for the mean male height, you should assign probability 0.95 to the proposition that the interval eventually generated will contain the true mean; but if you notice that the 10 men all happen to be over 6 feet tall, it becomes much less likely.

(See also my answer here.)

$\endgroup$
4
  • $\begingroup$ This seems like a Bayesian(-ish) way to make sense of a confidence interval, but I can't follow it. If I decide to use a 95% confidence interval, that is nothing to do with a prior probability. It's closer to what may seem like a quite different decision, to use a 95% point to characterise the tail of a univariate distribution. The decision in each case depends on convention and convenience. The repeated sampling interpretation of CIs can vary between fanciful and absurd, but it remains the main idea. I fear I am missing your point completely. $\endgroup$
    – Nick Cox
    May 14, 2021 at 13:32
  • $\begingroup$ @NickCox I made a minor edit but doubt it will resolve your objection. I think the connection between a 95% confidence interval and prior probability is that prior to seeing the data/interval, the probability that the interval will contain the parameter is 95%. $\endgroup$
    – fblundun
    May 14, 2021 at 14:37
  • $\begingroup$ Hmm. Thanks for the comment, but as said I can't see that you need to invoke a prior view, or that it helps in explaining. $\endgroup$
    – Nick Cox
    May 14, 2021 at 14:50
  • $\begingroup$ @NickCox any explanation of a 95% confidence interval has to include the proviso that the 0.95 probability only applies prior to seeing the interval - failure to account for this proviso is what causes probably the most common misunderstanding of CIs. $\endgroup$
    – fblundun
    May 14, 2021 at 15:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.