What, precisely, is a confidence interval?

Question

I know roughly and informally what a confidence interval is. However, I can't seem to wrap my head around one rather important detail: According to Wikipedia:

A confidence interval does not predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained.

I've also seen similar points made in several places on this site. A more correct definition, also from Wikipedia, is:

if confidence intervals are constructed across many separate data analyses of repeated (and possibly different) experiments, the proportion of such intervals that contain the true value of the parameter will approximately match the confidence level

Again, I've seen similar points made in several places on this site. I don't get it. If, under repeated experiments, the fraction of computed confidence intervals that contain the true parameter $\theta$ is $(1 - \alpha)$, then how can the probability that $\theta$ is in the confidence interval computed for the actual experiment be anything other than $(1 - \alpha)$? I'm looking for the following in an answer:

1. Clarification of the distinction between the incorrect and correct definitions above.

2. A formal, precise definition of a confidence interval that clearly shows why the first definition is wrong.

3. A concrete example of a case where the first definition is spectacularly wrong, even if the underlying model is correct.

Answer 1

I found this thought experiment helpful when thinking about confidence intervals. It also answers your question 3.

Let $X\sim U(0,1)$ and $Y=X+a-\frac{1}{2}$. Consider two observations of $Y$ taking the values $y_1$ and $y_2$ corresponding to observations $x_1$ and $x_2$ of $X$, and let $y_l=\min(y_1,y_2)$ and $y_u=\max(y_1,y_2)$. Then $[y_l,y_u]$ is a 50% confidence interval for $a$ (since the interval includes $a$ if $x_1<\frac12<x_2$ or $x_1>\frac12>x_2$, each of which has probability $\frac14$).

However, if $y_u-y_l>\frac12$ then we know that the probability that the interval contains $a$ is $1$, not $\frac12$. The subtlety is that a $z\%$ confidence interval for a parameter means that the endpoints of the interval (which are random variables) lie either side of the parameter with probability $z\%$ before you calculate the interval, not that the probability of the parameter lying within the interval is $z\%$ after you have calculated the interval.

Answer 2

There are many issues concerning confidence intervals, but let's focus on the quotations. The problem lies in possible misinterpretations rather than being a matter of correctness. When people say a "parameter has a particular probability of" something, they are thinking of the parameter as being a random variable. This is not the point of view of a (classical) confidence interval procedure, for which the random variable is the interval itself and the parameter is determined, not random, yet unknown. This is why such statements are frequently attacked.

Mathematically, if we let $t$ be any procedure that maps data $\mathbf{x} = (x_i)$ to subsets of the parameter space and if (no matter what the value of the parameter $\theta$ may be) the assertion $\theta \in t(\mathbf{x})$ defines an event $A(\mathbf{x})$, then--by definition--it has a probability $\Pr_{\theta}\left( A(\mathbf{x}) \right)$ for any possible value of $\theta$. When $t$ is a confidence interval procedure with confidence $1-\alpha$ then this probability is supposed to have an infimum (over all parameter values) of $1-\alpha$. (Subject to this criterion, we usually select procedures that optimize some additional property, such as producing short confidence intervals or symmetric ones, but that's a separate matter.) The Weak Law of Large Numbers then justifies the second quotation. That, however, is not a definition of confidence intervals: it is merely a property they have.

I think this analysis has answered question 1, shows that the premise of question 2 is incorrect, and makes question 3 moot.

Answer 3

I wouldn't call the definition of CIs as wrong, but they are easy to mis-interpret, due to there being more than one definition of probability. CIs are based on the following definition of Probability (Frequentist or ontological)

(1)probability of a proposition=long run proportion of times that proposition is observed to be true, conditional on the data generating process

Thus, in order to be conceptually valid in using a CI, you must accept this definition of probability. If you don't, then your interval is not a CI, from a theoretical point of view.

This is why the definition used the word proportion and NOT the word probability, to make it clear that the "long run frequency" definition of probability is being used.

The main alternative definition of Probability (Epistemological or probability as an extension of deductive Logic or Bayesian) is

(2)probability of a proposition = rational degree of belief that the proposition is true, conditional on a state of knowledge

People often intuitively get both of these definitions mixed up, and use whichever interpretation happens to appeal to their intuition. This can get you into all kinds of confusing situations (especially when you move from one paradigm to the other).

That the two approaches often lead to the same result, means that in some cases we have:

rational degree of belief that the proposition is true, conditional on a state of knowledge = long run proportion of times that proposition is observed to be true, conditional on the data generating process

The point is that it does not hold universally, so we cannot expect the two different definitions to always lead to the same results. So, unless you actually work out the Bayesian solution, and then find it to be the same interval, you cannot give the interval given by the CI the interpretation as a probability of containing the true value. And if you do, then the interval is not a Confidence Interval, but a Credible Interval.

Answer 4

R.A. Fisher had a criterion for the usefulness of confidence intervals: A CI should not admit of "identifiable subsets" that imply a different confidence level. In most (if not all) counterexamples, we have cases where there are identifiable subsets that have different coverage probabilities.

In theses cases, you can either use Bayesian cred-intervals to specify a subjective sense of where the parameter is, or you can formulate a likelihood interval to reflect the relative uncertainty in the parameter, given the data.

For example, one case that seems relatively contradiction-free is the 2-sided normal confidence interval for the population mean. Assuming sampling from a normal population with given std., the 95% CI admits of no identifiable subsets that would provide more information about the parameter. This can be seen by the fact that the sample mean is a sufficient statistic in the likelihood function - i.e., the likelihood function is independent of the individual sample values once we know the sample mean.

The reason we have any subjective confidence in the 95% symmetric CI for the normal mean stems less from the stated coverage probability and more from the fact that the symmetric 95% CI for the normal mean is the "highest likelihood" interval, i.e., all parameter values within the interval have a higher likelihood than any parameter value outside the interval. However, since likelihood is not a probability (in the long-run accuracy sense), it is more of a subjective criterion (as is the Bayesian use of prior and likelihood). In sum, there are infinitely many intervals for the normal mean that have 95% coverage probability, but only the symmetric CI has the intuitive plausbiltiy that we expect from an interval estimate.

Therefore, R.A. Fisher's criterion implies that coverage probability should equate with subjective confidence only if it admits of none of these identifiable subsets. If subsets are present, then the coverage probabilty will be conditional on the true values of the parameter(s) describing the subset. To get an interval with the intuitive level of confidence, you would need to condition the interval estiamte on the appropriate ancillary statistics that help identify the subset. OR, you could resort to dispersion/mixture models, which naturally leads to interpreting the parameters as random variables (aka Bayesian statistics) or you can calculate the profile/conditional/marginal likelihoods under the likelihood framework. Either way, you've abandoned any hope of coming up with an objectively verifiable probabilty of being correct, only a subjective "ordering of preferences."

Hope this helps.

Answer 5

This is thing that may be hard to understand:

• if on average 95% of all confidence intervals will contain the parameter
• and I have one specific confidence interval
• why isn't the the probability that this interval contains the parameter also 95% ?

A confidence interval relates to the sampling procedure. If you would take many samples and calculate a 95% confidence interval for each sample, you'd find that 95% of those intervals contain the population mean.

This is useful to for instance industrial quality departments. Those guys take many samples, and now they have the confidence that most of their estimates will be pretty close to the reality. They know that 95% of their estimates are pretty good, but they can't say that about each and every specific estimate.

Compare this to rolling dice: if you would roll 600 (fair) dice, how many 6 would you throw? Your best guess is $\frac{1}{6}$ * 600 = 100.

However, if you have thrown ONE die, it is useless to say: "There is a 1/6 or 16.6% probability that I have now thrown a 6". Why? Because the die shows either a 6, or some other figure. You have thrown a 6, or not. So the probability is 1, or 0. The probability cannot be $\frac{1}{6}$.

When asked before the throw what the probability of throwing a 6 with ONE die would be, a Bayesian would answer "$\frac{1}{6}$" (based on prior information: everybody knows that a die has 6 sides and an equal chance of falling on either of them), but a Frequentist would say "No idea" because frequentism is solely based on the data, not on priors or any outside information.

Likewise, if you have only 1 sample (thus 1 confidence interval), you have no way to say how likely it is that the population mean is in that interval. The mean (or any parameter) is either in it, or not. The probability is either 1, or 0.

Also, it is not correct that values within the Confidence Interval are more likely than those outside of that. I made a small illustration; everything is measured in °C. Remember, water freezes at 0 °C and boils at 100 °C.

The case: in a cold lake, we'd like to estimate the temperature of the water that flows below the ice. We measure the temperature in 100 locations. Here are my data:

• 0.1 °C (measured in 49 locations);
• 0.2 °C (also in 49 locations);
• 0 °C (in 1 location. This was water just about to freeze);
• 95 °C (in one location, there is a factory that illegally dumps very hot water in the lake).
• Mean temperature: 1.1 °C;
• Standard deviation: 1.5 °C;
• 95%-CI: ( -0.8 °C...... + 3.0 °C).

The temperatures within in this confidence interval are definitely NOT more likely than those outside of it. The average temperature of the flowing water in this lake CANNOT be colder than 0°C, otherwise it would not be water but ice. A part of this confidence interval (namely, the section from -0.8 to 0) actually has a 0% probability of containing the true parameter.

In conclusion: confidence intervals are a frequentist concept, and therefore are based on the idea of repeated samples. If many researchers would take samples from this lake, and if all those researchers would calculate confidence intervals, then 95% of those intervals will contain the true parameter. But for one single confidence interval it is impossible to say how likely it is that it contains the true parameter.

Answer 6

From a theoretical perspective Questions 2 and 3 are based on the incorrect assumption that the definitions are wrong. So I am in agreement with @whuber's answer in that respect, and @whuber's answer to question 1 does not require any additional input from me.

However, from a more practical perspective a confidence interval can be given its intuitive definition (Probability of containing the true value) when it is numerically identical with a Bayesian credible interval based on the same information (i.e. a non-informative prior).

But this is somewhat disheartening for the die hard anti-bayesian, because in order to verify the conditions to give his CI the interpretation he/she want to give it, they must work out the Bayesian solution, for which the intuitive interpretation automatically holds!

The easiest example is a $1-\alpha$ confidence interval for the normal mean with a known variance $\overline{x}\pm \sigma Z_{\alpha/2} $, and a $1-\alpha$ posterior credible interval $\overline{x}\pm \sigma Z_{\alpha/2} $.

I am not exactly sure of the conditions, but I know the following are important for the intuitive interpretation of CIs to hold:

1) a Pivot statistic exists, whose distribution is independent of the parameters (do exact pivots exist outside normal and chi-square distributions?)

2) there are no nuisance parameters, (except in the case of a Pivotal statistic, which is one of the few exact ways one has to handle nuisance parameters when making CIs)

3) a sufficient statistic exists for the parameter of interest, and the confidence interval uses the sufficient statistic

4) the sampling distribution of the sufficient statistic and the posterior distribution have some kind of symmetry between the sufficient statistic and the parameter. In the normal case the sampling distribution the symmetry is in $(\overline{x}|\mu,\sigma)\sim N(\mu,\frac{\sigma}{\sqrt{n}})$ while $(\mu|\overline{x},\sigma)\sim N(\overline{x},\frac{\sigma}{\sqrt{n}})$.

These conditions are usually difficult to find, and usually it is quicker to work out the Bayesian interval, and compare it. An interesting exercise may also be to try and answer the question "for what prior is my CI also a Credible Interval?" You may discover some hidden assumptions about your CI procedure by looking at this prior.

Answer 7

Here's the most degenerate example. If I want to build a 95% confidence interval $I$ for a real-valued parameter $\mu$, I can use the following distribution:

$$ \mathbb P(I = (-\infty, +\infty)) = 0.95 \\ \mathbb P(I = \emptyset) = 0.05 $$

(Some definitions of confidence interval may not technically include infinite or empty intervals, but this doesn't affect the example). 95% of confidence intervals drawn from this distribution will contain $\mu$. But if I show you any particular confidence interval drawn from this distribution, your probability that it contains $\mu$ shouldn't be 0.95. It should be 1 for $(-\infty, +\infty)$ and 0 for $\emptyset$.

Answer 8

Suppose we are in a simple situation. You have an unknown parameter $\theta$ and $T$ an estimator of $\theta$ that has an imprecision around 1 (informally). You think (informally) $\theta$ should be in $[T-1;T+1]$ most often.

In a real experiment you observe $T=12$.

It is natural to ask the question "Given what I see ($T=12$), what is the probability $\theta\in[11;13]$ ?". Mathematically : $P(\theta\in[11;13]|T=12)$. Everybody naturally asks this question. The confidence interval theory should logically answer to this question. But it doesn't.

Bayesian statistics do answer to that question. In Bayesian statistic, you can really calculate $P(\theta\in[11;13]|T=12)$. But you need to assume a prior that is a distribution for $\theta$ before doing the experiment and observing $T$. For example :

• Assume $\theta$ has a prior distribution uniform on $[0;30]$
• do this experiment, find $T=12$
• Apply Bayes formula : $P(\theta\in[11;13]|T=12)=0.94$

But in frequentist statistics, there is no prior and thus anything like $P(\theta\in...|T \in...)$ does not exist. Instead statisticians say something like this : "Whatever $\theta$ is, the probability that $\theta\in [T-1;T+1]$ is $0.95$". Mathematically : $\forall\theta, P(\theta\in[T-1;T+1]|\theta)=0.95$"

So :

• Bayesian : $P(\theta\in[T-1;T+1]|T)=0.94$ for $T=12$
• Frequentist : $\forall\theta, P(\theta\in[T-1;T+1]|\theta)=0.95$

The Bayesian statement is more natural. Most often, the frequentist statement is misinterpreted spontaneously as the Bayesian statement (by any normal human brain who hasn't practised statistics for years). And honestly, many statistics book do not make that point very clear.

And practically ?

In many usual situations the fact is that probabilities obtained by frequentist and Bayesian approaches are very close. So that confusing the frequentist statement for the Bayesian one has little consequences. But "philosophically" it's very different.

Answer 9

Okay, I realize that when you calculate a 95% confidence interval for a parameter using classical frequentist methods, it doesn't mean that there is a 95% probability that the parameter lies within that interval. And yet ... when you approach the problem from a Bayesian perspective, and calculate a 95% credible interval for the parameter, you get (assuming a non-informative prior) exactly the same interval that you get using the classical approach. So, if I use classical statistics to calculate the 95% confidence interval for (say) the mean of a data set, then it is true that there's a 95% probability that the parameter lies in that interval.

Answer 10

You are asking about the Frequentist confidence interval. The definition (note that none of your 2 citation is a definition! Just statements, which both are correct) is:

If I had repeated this experiment a big number of times, given this fitted model with this parameter values, in 95% of experiments the estimated value of a parameter would fall within this interval.

So you have a model (built using your observed data) and its estimated parameters. Then if you generated some hypothetical data sets according to this model and parameters, the estimated parameters would fall inside the confidence interval.

So in fact, this frequentist approach takes the model and estimated parameters as fixed, as given, and treats your data as uncertain - as a random sample of many many other possible data.

This is really hard to interpret and this is often used as an argument for Bayesian statistics (which I think can be sometimes little disputable. The bayesian statistics on the other hand takes your data as fixed and treats parameters as uncertain. The bayesian credible intervals are then actually intuitive, as you'd expect: bayesian credible intervals are intervals where with 95% the real parameter value lies.

But in practice many people interpret the frequentist confidence intervals in the same way as Bayesian credible intervals and many statisticians don't consider this a big issue - though they all know, it is not 100% correct. Also in practice, the frequentist and bayesian confidence/credible intervals won't differ much, when using bayesian uninformative priors.

Answer 11

1. Suppose that we want to study the height of men $X$ in Canada for the last 2 years. Assume also that $X \sim N(\mu,\sigma^2)$ with $\sigma^2$ known. Then, talking about the probability \begin{equation*} P(X>1.80)=P(\omega \in \Omega:X(\omega)>1.80) \end{equation*} that a random man from Canada has a height more than $1.80$ makes sense, because $X$ is a random variable. On the other hand, talking about the probability \begin{equation*} \hspace{10mm} P(\mu>1.80)=P(\omega\in \Omega:\mu(\omega)>1.80) \hspace{4mm}(?) \end{equation*} that the average height of men in Canada is more than $1.80$ does not make sense, because the actual parameter is equal to a fixed number and it is not a random variable (if for example we knew that $\mu=1.75$, then $P(\mu>1.80)=0$, because $\mu=1.75$). The average height of men in Canada for the last 2 years is a unique number; it is not possible that the average height of any population takes more than one value.

2. Similarly, if $\{X_1,...,X_n\}$ are i.i.d. random variables from $X$, then $\overline{X}_n=\frac{X_1+...+X_n}{n}$ is also a random variable, so talking about the probability \begin{equation*} P\Big(\overline{X}_n \in \Big[\mu-z_{a/2}\frac{\sigma}{\sqrt{n}},\mu+z_{a/2}\frac{\sigma}{\sqrt{n}}\Big ] \Big)=0.95 \end{equation*} makes sence, while talking about the probability \begin{equation*} P\Big(\mu \in \Big[\overline{X}_n-z_{a/2}\frac{\sigma}{\sqrt{n}},\overline{X}_n+z_{a/2}\frac{\sigma}{\sqrt{n}}\Big ] \Big) \end{equation*} does not, because $\mu$ is not a random variable (we could only say that the last probability is either 1 or 0, because $\mu$ is either contained in the interval or not).

3. However, the probability \begin{equation*} P\Big(\overline{X}_n \in \Big[\mu-z_{a/2}\frac{\sigma}{\sqrt{n}},\mu+z_{a/2}\frac{\sigma}{\sqrt{n}}\Big ] \Big)=0.95 \end{equation*} means that, if we take a random sample $\{x_1,...,x_n\}$ of heights many times and estimate the empirical average height $\overline{x}=\frac{x_1+...+x_n}{n}$ each time, then $\overline{x}$ will be contained in the intervals \begin{equation*} \overline{x} \in \Big[\mu -z_{a/2}\frac{\sigma}{\sqrt{n}},\mu+z_{a/2}\frac{\sigma}{\sqrt{n}}\Big ] \end{equation*} approximately $95\%$ of these times. Equivalently, taking a random sample $\{x_1,...,x_n\}$ from $X$ many times, $\mu$ will be contained in the intervals \begin{equation*} \mu \in \Big[\overline{x} -z_{a/2}\frac{\sigma}{\sqrt{n}},\overline{x}+z_{a/2}\frac{\sigma}{\sqrt{n}}\Big ] \end{equation*} $95\%$ of these times.

4. Note that, each time we take a different sample $\{x_1,...,x_n\}$ from $X$, the confidence interval changes, since its center $\overline{x}=\frac{x_1+...+x_n}{n}$ does. However, the percentage of times that $\mu$ is contained in these different confidence intervals will still be approximately $95 \%$.

Answer 12

A "confidence interval" is a specific case of the broader concept of a "confidence set", which may or may not be a single connected interval. The broader concept can be conceived mathematically as follows. Suppose we have an observable data vector $\mathbf{X}_n \equiv (X_1,...,X_n)$ from a distribution with unknown parameter $\theta \in \Theta$. Then a confidence set for the parameter $\theta$ is created from a set function $\mathcal{S}$ that satisfies the following conditional probability requirement:$^\dagger$

$$1-\alpha = \mathbb{P}(\theta \in \mathcal{S}(\mathbf{X}_n, \alpha)|\theta) \quad \quad \quad \text{for all } \theta \in \Theta \text{ and } 0 \leqslant \alpha \leqslant 1.$$

Note that if $\theta$ is conceived as a random variable then this requirement also implies the following weaker property pertaining to the marginal probability of inclusion:

$$1-\alpha = \mathbb{P}(\theta \in \mathcal{S}(\mathbf{X}_n, \alpha)) \quad \quad \quad \text{for all } 0 \leqslant \alpha \leqslant 1. \quad \quad \quad \quad$$

Now, given a set function that comports to the above conditional probability requirement, for a given value $0 \leqslant \alpha \leqslant 1$, the confidence set for the data $\mathbf{x}_n$ (with confidence level $1-\alpha$) is the fixed set $\mathcal{S}(\mathbf{x}_n, \alpha)$. In the case where this is a single connected interval, we call it a confidence interval.

As can be seen, the confidence set is a fixed set determined by the observed data. As such, it is not possible to make any non-degenerate probability statement about its coverage of a fixed parameter. However, if we treat the data as random, we can see that the random confidence set will contain the conditioning value of the parameter $\theta$ with probability equal to the confidence level. This holds regardless of the conditioning value, and so it also holds as a marginal property if $\theta$ is a random variable. As discussed in a related answer, this is an extremely useful and robust property.

---

$^\dagger$ One slight complication here is that we sometimes form confidence intervals that only satisfy this probability requirement under some approximating assumption (e.g., distributional approximation using the central limit theorem). In these cases the probability requirement is satisfied exactly under some simplifying assumption and it holds approximately in broader cases.