I know roughly and informally what a confidence interval is. However, I can't seem to wrap my head around one rather important detail: According to Wikipedia:

A confidence interval does not predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained.

I've also seen similar points made in several places on this site. A more correct definition, also from Wikipedia, is:

if confidence intervals are constructed across many separate data analyses of repeated (and possibly different) experiments, the proportion of such intervals that contain the true value of the parameter will approximately match the confidence level

Again, I've seen similar points made in several places on this site. I don't get it. If, under repeated experiments, the fraction of computed confidence intervals that contain the true parameter $\theta$ is $(1 - \alpha)$, then how can the probability that $\theta$ is in the confidence interval computed for the actual experiment be anything other than $(1 - \alpha)$? I'm looking for the following in an answer:

  1. Clarification of the distinction between the incorrect and correct definitions above.

  2. A formal, precise definition of a confidence interval that clearly shows why the first definition is wrong.

  3. A concrete example of a case where the first definition is spectacularly wrong, even if the underlying model is correct.

  • 4
    $\begingroup$ This post has some good discussion of the issue of confidence intervals stats.stackexchange.com/questions/2356/… . The article referred to in the post, I think, helps shed some light one precisely why the above definitions are correct for confidence intervals. It is often when viewing how CIs break down that one is able to understand them better. $\endgroup$ – probabilityislogic Jan 28 '11 at 3:21
  • 2
    $\begingroup$ Part of me applauds the question (+1). A competing part wants to point out that 1. The vast majority of statistics consumers, people who use statistics pragmatically but not philosophically in order to make their point in chemistry or market research, will never grasp the niceties of the issues, and we will often be at a loss to explain results. 2. Even some purist statisticians can fall into the trap of making supposedly probabilistic statements like those involving confidence intervals when they are not working with random samples. A much bigger issue. $\endgroup$ – rolando2 Feb 1 '11 at 1:33
  • 4
    $\begingroup$ @Mario Your assumption is not true! Out of 100 repetitions of the experiment, we expect 95 of the CIs (not the means) to contain the true (but unknown) mean. The CI is random but the true population mean is not. $\endgroup$ – whuber May 12 '11 at 4:01
  • 8
    $\begingroup$ There is a nice paper by Cumming & Maillardet (2006) showing that not 95% of replication means will fall into the original CI, but only 83.4% (they call this value 'capture percentage'). The reason is that there are two sources of variability: A) the variability of the original mean around mu, and, B) the variability of replication means around mu. Most people forget A: the original CI is not necessarliy constructed around mu! $\endgroup$ – Felix S Sep 2 '11 at 7:13
  • 3
    $\begingroup$ Interested readers may also want to see this thread: Why does a 95% CI not imply a 95% chance of containing the mean? $\endgroup$ – gung - Reinstate Monica Jan 8 '13 at 16:04

11 Answers 11


I found this thought experiment helpful when thinking about confidence intervals. It also answers your question 3.

Let $X\sim U(0,1)$ and $Y=X+a-\frac{1}{2}$. Consider two observations of $Y$ taking the values $y_1$ and $y_2$ corresponding to observations $x_1$ and $x_2$ of $X$, and let $y_l=\min(y_1,y_2)$ and $y_u=\max(y_1,y_2)$. Then $[y_l,y_u]$ is a 50% confidence interval for $a$ (since the interval includes $a$ if $x_1<\frac12<x_2$ or $x_1>\frac12>x_2$, each of which has probability $\frac14$).

However, if $y_u-y_l>\frac12$ then we know that the probability that the interval contains $a$ is $1$, not $\frac12$. The subtlety is that a $z\%$ confidence interval for a parameter means that the endpoints of the interval (which are random variables) lie either side of the parameter with probability $z\%$ before you calculate the interval, not that the probability of the parameter lying within the interval is $z\%$ after you have calculated the interval.

  • 3
    $\begingroup$ Note that $Y>a$ almost surely, hence the interval $[y_l,y_u]$ contains the parameter $a$ with probability zero. In fact your argument works if what you are estimating is $\theta=a+\frac12$. $\endgroup$ – Did Feb 2 '11 at 13:58
  • 4
    $\begingroup$ I don't think this counter example is valid, becuase you only know the probability that the interval contains $\theta$ is one after seeing that $y_u-y_l>1/2$. It is perfectly reasonable that the probability should change after we acquire additional information. If all you knew was that the interval was a 50% confidence interval, then the probability would still be 1/2 (although it would be a Bayesian probability not a frequentist one as it applies to a particular event which has no long run frequency) $\endgroup$ – Dikran Marsupial Jun 9 '11 at 11:00
  • 1
    $\begingroup$ That is indeed a good example, but I strongly disagree with your statements about probabilities somehow changing before and after calculating the confidence interval. That makes no sense, and gives the impression that the math somehow cares about what you know and what you don't. It doesn't!!. You always have that $\mathbb{P}(a \in [y_l,y_u])$ is $\tfrac{1}{2}$. You also always have that $\mathbb{P}(a \in [y_l,y_u] \;|\; y_u - y_l > \tfrac{1}{2})$ is $1$. That's not a contradiction, one is simply an unconditional probability and the other is a conditional probability. $\endgroup$ – fgp Aug 7 '16 at 17:52
  • 2
    $\begingroup$ @fgp, yes, perhaps it's poor wording on Taylor's part talking about probabilities changing. No probabilities are changing. What the argument is showing is how it's easy for situations to arise demonstrating mistaken understanding of CIs lead to logical problems. If you believe a CI you observe has a 50% probability of being correct but it can't possibly be correct then you're understanding of a CI is wrong. $\endgroup$ – John Feb 1 '17 at 13:12

There are many issues concerning confidence intervals, but let's focus on the quotations. The problem lies in possible misinterpretations rather than being a matter of correctness. When people say a "parameter has a particular probability of" something, they are thinking of the parameter as being a random variable. This is not the point of view of a (classical) confidence interval procedure, for which the random variable is the interval itself and the parameter is determined, not random, yet unknown. This is why such statements are frequently attacked.

Mathematically, if we let $t$ be any procedure that maps data $\mathbf{x} = (x_i)$ to subsets of the parameter space and if (no matter what the value of the parameter $\theta$ may be) the assertion $\theta \in t(\mathbf{x})$ defines an event $A(\mathbf{x})$, then--by definition--it has a probability $\Pr_{\theta}\left( A(\mathbf{x}) \right)$ for any possible value of $\theta$. When $t$ is a confidence interval procedure with confidence $1-\alpha$ then this probability is supposed to have an infimum (over all parameter values) of $1-\alpha$. (Subject to this criterion, we usually select procedures that optimize some additional property, such as producing short confidence intervals or symmetric ones, but that's a separate matter.) The Weak Law of Large Numbers then justifies the second quotation. That, however, is not a definition of confidence intervals: it is merely a property they have.

I think this analysis has answered question 1, shows that the premise of question 2 is incorrect, and makes question 3 moot.

  • 3
    $\begingroup$ Thank you for providing an answer to an excellent question. May I bring up the following analogy for further discussion? Suppose I flip a fair coin over and over. Then, $P(Head) = .50$. Now, I flip the coin once, but don't show you what I flipped, and I ask: "What is the probability that heads are up?". How would you answer that question? $\endgroup$ – Wolfgang Jan 28 '11 at 12:46
  • 4
    $\begingroup$ Another way to phrase it: for non-Bayesians, the only "things" that can have a probability are possible events - in the sense of future outcomes of a random experiment. Given that the parameter has a fixed true value, once you have an interval with specific values, it's not a possible event anymore whether or not the parameter is included in the interval. As a result, you can have confidence in the process that generates the interval, but not in two specific numbers. $\endgroup$ – caracal Jan 28 '11 at 13:50
  • 1
    $\begingroup$ @caracal - just some food for thought, is a "coin flip" every truly "random"? If you say "yes" then you would reject the idea that whether a coin comes up heads is a deterministic (but complicated) function of many things (say- wind,altitude,force and angle of flip,weight of coin,etc.etc.). I think this shows the double standard of "randomness" that applies to CI-based thinking, The data are fixed but we are uncertain about its value (ergo data are random), while the parameters are fixed but we are uncertain about its value (ergo parameters are not random). $\endgroup$ – probabilityislogic Jan 28 '11 at 14:33
  • 4
    $\begingroup$ @Wolfgang I do not see how your example pertains to confidence intervals. You do not ask for anything related to a distributional parameter. Your situation is most closely related to prediction intervals. I think this entire discussion may have some interest in that context, but it doesn't belong in a thread about confidence intervals. $\endgroup$ – whuber Jan 28 '11 at 16:36
  • 2
    $\begingroup$ @whuber The question whether one can make a probability statement about a particular 95% CI capturing the true unknown parameter is very similar to the question whether one can make a probability statement about a specific flip where the outcome is still unknown. In the long run, 95% of the CIs will capture the parameter. In the long run, 50% of the flips are heads. Can we say that there is a 95% chance that a particular CI captures the parameter? Can we say that there is a 50% chance that heads are up before looking? I would say yes to both. But some people may disagree. $\endgroup$ – Wolfgang Jan 28 '11 at 16:55

I wouldn't call the definition of CIs as wrong, but they are easy to mis-interpret, due to there being more than one definition of probability. CIs are based on the following definition of Probability (Frequentist or ontological)

(1)probability of a proposition=long run proportion of times that proposition is observed to be true, conditional on the data generating process

Thus, in order to be conceptually valid in using a CI, you must accept this definition of probability. If you don't, then your interval is not a CI, from a theoretical point of view.

This is why the definition used the word proportion and NOT the word probability, to make it clear that the "long run frequency" definition of probability is being used.

The main alternative definition of Probability (Epistemological or probability as an extension of deductive Logic or Bayesian) is

(2)probability of a proposition = rational degree of belief that the proposition is true, conditional on a state of knowledge

People often intuitively get both of these definitions mixed up, and use whichever interpretation happens to appeal to their intuition. This can get you into all kinds of confusing situations (especially when you move from one paradigm to the other).

That the two approaches often lead to the same result, means that in some cases we have:

rational degree of belief that the proposition is true, conditional on a state of knowledge = long run proportion of times that proposition is observed to be true, conditional on the data generating process

The point is that it does not hold universally, so we cannot expect the two different definitions to always lead to the same results. So, unless you actually work out the Bayesian solution, and then find it to be the same interval, you cannot give the interval given by the CI the interpretation as a probability of containing the true value. And if you do, then the interval is not a Confidence Interval, but a Credible Interval.

  • 2
    $\begingroup$ I do not see why the probability of a proposition according to definition 1 should be a rational number. Long run proportion seems to refer to the limit of the proportions of times such that the proposition is observed to be true. Each proportion is a rational number but their limit might not be. (Fortunately, this parenthesis of yours seems tangential at best to the rest of your answer.) $\endgroup$ – Did Jan 31 '11 at 7:55
  • 3
    $\begingroup$ @probability This reply seems to be taking us off on a tangent in a not very constructive way. Equating probability and proportion is a form of ontological confusion, akin to equating a temperature with the level of mercury in a thermometer: one is a theoretical construct and the other is a physical phenomenon used to measure it. There is some discussion of this at stats.stackexchange.com/questions/1525/… . $\endgroup$ – whuber Jan 31 '11 at 15:02
  • $\begingroup$ @Didier - you're right, in fact the sequence of $x_n=\frac{r}{2x_{n-1}}+\frac{x_{n-1}}{2} \rightarrow \sqrt{r}$, which is rational terms with irrational limit. I have removed this remark. Thanks for bringing this up. $\endgroup$ – probabilityislogic Jan 31 '11 at 21:03
  • 6
    $\begingroup$ @whuber - The point is relevant to bring up because it is precisely this misunderstanding that leads people to interpret CIs in the wrong way. Confusing probability with "rational degree of belief" is not consistent with the frequentist paradigm. This is what happens when you take CIs to mean "probability of true value being in interval", which is what @dsimcha is doing in the question. $\endgroup$ – probabilityislogic Jan 31 '11 at 21:10
  • 1
    $\begingroup$ @probability Thank you for the explanation. I had understood your reply as being in accord with a definition of "probability = proportion." In fact, a close rereading still suggests this is what you are saying in the third paragraph, even though your comment now characterizes this as a misunderstanding. You might want to clarify this point. $\endgroup$ – whuber Jan 31 '11 at 21:42

R.A. Fisher had a criterion for the usefulness of confidence intervals: A CI should not admit of "identifiable subsets" that imply a different confidence level. In most (if not all) counterexamples, we have cases where there are identifiable subsets that have different coverage probabilities.

In theses cases, you can either use Bayesian cred-intervals to specify a subjective sense of where the parameter is, or you can formulate a likelihood interval to reflect the relative uncertainty in the parameter, given the data.

For example, one case that seems relatively contradiction-free is the 2-sided normal confidence interval for the population mean. Assuming sampling from a normal population with given std., the 95% CI admits of no identifiable subsets that would provide more information about the parameter. This can be seen by the fact that the sample mean is a sufficient statistic in the likelihood function - i.e., the likelihood function is independent of the individual sample values once we know the sample mean.

The reason we have any subjective confidence in the 95% symmetric CI for the normal mean stems less from the stated coverage probability and more from the fact that the symmetric 95% CI for the normal mean is the "highest likelihood" interval, i.e., all parameter values within the interval have a higher likelihood than any parameter value outside the interval. However, since likelihood is not a probability (in the long-run accuracy sense), it is more of a subjective criterion (as is the Bayesian use of prior and likelihood). In sum, there are infinitely many intervals for the normal mean that have 95% coverage probability, but only the symmetric CI has the intuitive plausbiltiy that we expect from an interval estimate.

Therefore, R.A. Fisher's criterion implies that coverage probability should equate with subjective confidence only if it admits of none of these identifiable subsets. If subsets are present, then the coverage probabilty will be conditional on the true values of the parameter(s) describing the subset. To get an interval with the intuitive level of confidence, you would need to condition the interval estiamte on the appropriate ancillary statistics that help identify the subset. OR, you could resort to dispersion/mixture models, which naturally leads to interpreting the parameters as random variables (aka Bayesian statistics) or you can calculate the profile/conditional/marginal likelihoods under the likelihood framework. Either way, you've abandoned any hope of coming up with an objectively verifiable probabilty of being correct, only a subjective "ordering of preferences."

Hope this helps.

  • 1
    $\begingroup$ (+1) One way to justify the symmetric Normal CI is that it minimizes expected length. Ultimately that just pushes back the subjectivity to the choice of length as a loss function in a decision procedure: but that's arguably a "good" kind of subjectivity (because it exposes the role of our analytical objectives in our choice of statistical procedure) rather than "bad" subjectivity, which sounds merely like some pejorative epithet. $\endgroup$ – whuber Mar 13 '12 at 18:14

From a theoretical perspective Questions 2 and 3 are based on the incorrect assumption that the definitions are wrong. So I am in agreement with @whuber's answer in that respect, and @whuber's answer to question 1 does not require any additional input from me.

However, from a more practical perspective a confidence interval can be given its intuitive definition (Probability of containing the true value) when it is numerically identical with a Bayesian credible interval based on the same information (i.e. a non-informative prior).

But this is somewhat disheartening for the die hard anti-bayesian, because in order to verify the conditions to give his CI the interpretation he/she want to give it, they must work out the Bayesian solution, for which the intuitive interpretation automatically holds!

The easiest example is a $1-\alpha$ confidence interval for the normal mean with a known variance $\overline{x}\pm \sigma Z_{\alpha/2} $, and a $1-\alpha$ posterior credible interval $\overline{x}\pm \sigma Z_{\alpha/2} $.

I am not exactly sure of the conditions, but I know the following are important for the intuitive interpretation of CIs to hold:

1) a Pivot statistic exists, whose distribution is independent of the parameters (do exact pivots exist outside normal and chi-square distributions?)

2) there are no nuisance parameters, (except in the case of a Pivotal statistic, which is one of the few exact ways one has to handle nuisance parameters when making CIs)

3) a sufficient statistic exists for the parameter of interest, and the confidence interval uses the sufficient statistic

4) the sampling distribution of the sufficient statistic and the posterior distribution have some kind of symmetry between the sufficient statistic and the parameter. In the normal case the sampling distribution the symmetry is in $(\overline{x}|\mu,\sigma)\sim N(\mu,\frac{\sigma}{\sqrt{n}})$ while $(\mu|\overline{x},\sigma)\sim N(\overline{x},\frac{\sigma}{\sqrt{n}})$.

These conditions are usually difficult to find, and usually it is quicker to work out the Bayesian interval, and compare it. An interesting exercise may also be to try and answer the question "for what prior is my CI also a Credible Interval?" You may discover some hidden assumptions about your CI procedure by looking at this prior.

  • 1
    $\begingroup$ (+1) Is there really such a person as an "anti-Bayesian"? :-) $\endgroup$ – whuber Mar 13 '12 at 18:11
  • 6
    $\begingroup$ @whuber Here's one. And here's an econometrician who collaborates with her on scholarship in philosophy of statistic. $\endgroup$ – Cyan Mar 15 '12 at 19:30
  • 1
    $\begingroup$ Thanks! That's an extremely interesting thread in the philosophy of probability and statistics of which I was unaware. $\endgroup$ – whuber Mar 15 '12 at 20:28
  • 1
    $\begingroup$ Did you miswrite $\overline x \pm \frac{z_{\alpha/2}\sigma}{\sqrt{n}}$ as missing $\sqrt{n}$? $\endgroup$ – qazwsx Nov 21 '15 at 23:05

This is thing that may be hard to understand:

  • if on average 95% of all confidence intervals will contain the parameter
  • and I have one specific confidence interval
  • why isn't the the probability that this interval contains the parameter also 95% ?

A confidence interval relates to the sampling procedure. If you would take many samples and calculate a 95% confidence interval for each sample, you'd find that 95% of those intervals contain the population mean.

This is useful to for instance industrial quality departments. Those guys take many samples, and now they have the confidence that most of their estimates will be pretty close to the reality. They know that 95% of their estimates are pretty good, but they can't say that about each and every specific estimate.

Compare this to rolling dice: if you would roll 600 (fair) dice, how many 6 would you throw? Your best guess is $\frac{1}{6}$ * 600 = 100.

However, if you have thrown ONE die, it is useless to say: "There is a 1/6 or 16.6% probability that I have now thrown a 6". Why? Because the die shows either a 6, or some other figure. You have thrown a 6, or not. So the probability is 1, or 0. The probability cannot be $\frac{1}{6}$.

When asked before the throw what the probability of throwing a 6 with ONE die would be, a Bayesian would answer "$\frac{1}{6}$" (based on prior information: everybody knows that a die has 6 sides and an equal chance of falling on either of them), but a Frequentist would say "No idea" because frequentism is solely based on the data, not on priors or any outside information.

Likewise, if you have only 1 sample (thus 1 confidence interval), you have no way to say how likely it is that the population mean is in that interval. The mean (or any parameter) is either in it, or not. The probability is either 1, or 0.

Also, it is not correct that values within the Confidence Interval are more likely than those outside of that. I made a small illustration; everything is measured in °C. Remember, water freezes at 0 °C and boils at 100 °C.

The case: in a cold lake, we'd like to estimate the temperature of the water that flows below the ice. We measure the temperature in 100 locations. Here are my data:

  • 0.1 °C (measured in 49 locations);
  • 0.2 °C (also in 49 locations);
  • 0 °C (in 1 location. This was water just about to freeze);
  • 95 °C (in one location, there is a factory that illegally dumps very hot water in the lake).
  • Mean temperature: 1.1 °C;
  • Standard deviation: 1.5 °C;
  • 95%-CI: ( -0.8 °C...... + 3.0 °C).

The temperatures within in this confidence interval are definitely NOT more likely than those outside of it. The average temperature of the flowing water in this lake CANNOT be colder than 0°C, otherwise it would not be water but ice. A part of this confidence interval (namely, the section from -0.8 to 0) actually has a 0% probability of containing the true parameter.

In conclusion: confidence intervals are a frequentist concept, and therefore are based on the idea of repeated samples. If many researchers would take samples from this lake, and if all those researchers would calculate confidence intervals, then 95% of those intervals will contain the true parameter. But for one single confidence interval it is impossible to say how likely it is that it contains the true parameter.

  • 1
    $\begingroup$ Don't confuse the fact that the frequentist statistic doesn't measure belief with a frequentist person having prior beliefs and updating them. The difference is not whether the frequentist is an idiot with no knowledge outside the data but whether the frequentist statistics provide direct measures of the belief states. The frequentist must update their beliefs based on tests, CIs, etc. otherwise their whole system doesn't work because everything depends on the decisions made. $\endgroup$ – John Feb 1 '17 at 13:03

Suppose we are in a simple situation. You have an unknown parameter $\theta$ and $T$ an estimator of $\theta$ that has an imprecision around 1 (informally). You think (informally) $\theta$ should be in $[T-1;T+1]$ most often.

In a real experiment you observe $T=12$.

It is natural to ask the question "Given what I see ($T=12$), what is the probability $\theta\in[11;13]$ ?". Mathematically : $P(\theta\in[11;13]|T=12)$. Everybody naturally asks this question. The confidence interval theory should logically answer to this question. But it doesn't.

Bayesian statistics do answer to that question. In Bayesian statistic, you can really calculate $P(\theta\in[11;13]|T=12)$. But you need to assume a prior that is a distribution for $\theta$ before doing the experiment and observing $T$. For example :

  • Assume $\theta$ has a prior distribution uniform on $[0;30]$
  • do this experiment, find $T=12$
  • Apply Bayes formula : $P(\theta\in[11;13]|T=12)=0.94$

But in frequentist statistics, there is no prior and thus anything like $P(\theta\in...|T \in...)$ does not exist. Instead statisticians say something like this : "Whatever $\theta$ is, the probability that $\theta\in [T-1;T+1]$ is $0.95$". Mathematically : $\forall\theta, P(\theta\in[T-1;T+1]|\theta)=0.95$"

So :

  • Bayesian : $P(\theta\in[T-1;T+1]|T)=0.94$ for $T=12$
  • Frequentist : $\forall\theta, P(\theta\in[T-1;T+1]|\theta)=0.95$

The Bayesian statement is more natural. Most often, the frequentist statement is misinterpreted spontaneously as the Bayesian statement (by any normal human brain who hasn't practised statistics for years). And honestly, many statistics book do not make that point very clear.

And practically ?

In many usual situations the fact is that probabilities obtained by frequentist and Bayesian approaches are very close. So that confusing the frequentist statement for the Bayesian one has little consequences. But "philosophically" it's very different.


Okay, I realize that when you calculate a 95% confidence interval for a parameter using classical frequentist methods, it doesn't mean that there is a 95% probability that the parameter lies within that interval. And yet ... when you approach the problem from a Bayesian perspective, and calculate a 95% credible interval for the parameter, you get (assuming a non-informative prior) exactly the same interval that you get using the classical approach. So, if I use classical statistics to calculate the 95% confidence interval for (say) the mean of a data set, then it is true that there's a 95% probability that the parameter lies in that interval.

  • 5
    $\begingroup$ Whether you get the same result using frequentist confidence intervals and Bayesian credible intervals depends on the problem, and in particular on the prior distribution used in the Bayesian approach. It also important in maths and science that when you are right you are right for the correct reason! $\endgroup$ – Dikran Marsupial Jan 28 '11 at 17:20
  • 4
    $\begingroup$ If you "use classical statistics to calculate the 95% confidence interval for [a parameter]," then, if you are reasoning consistently, it is meaningless to refer to a "probability that the parameter lies in that interval." The moment you mention that probability, you have changed your statistical model of the situation. In the new model, where the parameter is random, it is incorrect to compute a CI using frequentist methods. Obtaining the right answer this way in some situations is interesting but does not justify the conceptual confusion that underlies it. $\endgroup$ – whuber Jan 28 '11 at 18:13
  • 4
    $\begingroup$ @whuber - your premise "...if you are reasoning consistently..." has a consequence from the good old Cox's theorem. It says that if you are reasoning consistently, then your solution must be mathematically equivalent to a Bayesian one. So, given this premise, a CI will necessarily be equivalent to a credible interval, and its interpretation as a probability is a valid one. And in Bayes, it is not the parameter which has a distribution, it is the uncertainty about that parameter which has a distribution. $\endgroup$ – probabilityislogic Jan 29 '11 at 17:58
  • 2
    $\begingroup$ ...cont'd...So one can play the silly game of I'm a Bayesian "Prob that parameter is in the interval", I'm a frequentist "prob that interval covers parameter", I'm a Bayesian..., I'm a frequentist,..., I'm a Bayesian..., I'm a frequentist,..... all the while the numbers of the actual calculation never change $\endgroup$ – probabilityislogic Jan 29 '11 at 18:01

You are asking about the Frequentist confidence interval. The definition (note that none of your 2 citation is a definition! Just statements, which both are correct) is:

If I had repeated this experiment a big number of times, given this fitted model with this parameter values, in 95% of experiments the estimated value of a parameter would fall within this interval.

So you have a model (built using your observed data) and its estimated parameters. Then if you generated some hypothetical data sets according to this model and parameters, the estimated parameters would fall inside the confidence interval.

So in fact, this frequentist approach takes the model and estimated parameters as fixed, as given, and treats your data as uncertain - as a random sample of many many other possible data.

This is really hard to interpret and this is often used as an argument for Bayesian statistics (which I think can be sometimes little disputable. The bayesian statistics on the other hand takes your data as fixed and treats parameters as uncertain. The bayesian credible intervals are then actually intuitive, as you'd expect: bayesian credible intervals are intervals where with 95% the real parameter value lies.

But in practice many people interpret the frequentist confidence intervals in the same way as Bayesian credible intervals and many statisticians don't consider this a big issue - though they all know, it is not 100% correct. Also in practice, the frequentist and bayesian confidence/credible intervals won't differ much, when using bayesian uninformative priors.

  • $\begingroup$ -1 Your "definition" appears to be incorrect, at least in one reading of it. The $1-\alpha$ CI is constructed to cover the true parameter with probability $1-\alpha$. It is not conditional on a particular model or method of fitting the parameters. Perhaps I am misreading the definition, though: I take "this fitted model with this parameter value" to refer to your current estimate of the parameter. If that's not how you intended it, perhaps you could clarify this point? $\endgroup$ – whuber Feb 17 '12 at 15:13
  • $\begingroup$ @whuber, OK, I take it, but if you say my definiton is wrong, please post your full definition of what CI is. $\endgroup$ – Curious Feb 17 '12 at 15:29
  • $\begingroup$ I have clarified my comment, Tomas, because it occurs to me I might be reading your definition in a way you did not intend. Kiefer, Introduction to Statistical Inference, writes "[T]he outcome of the experiment is $X$...[S]uppose the procedure $t=[L,U]$ is used to estimate $\phi(\theta)$ and the true value of $\theta$ is $\theta_0$...[T]he quantity $\gamma_t(\theta_0)=\Pr_{\theta_0}\{L(X)\le \phi(\theta_0)\le U(X)\}$...The number $\bar{\gamma}_t=\inf_{\theta\in\Omega}\gamma_t(\theta)$ is called the confidence coefficient of the procedure $t$...$t$ is called a confidence interval." $\endgroup$ – whuber Feb 17 '12 at 15:40
  • $\begingroup$ @whuber, your definition is really uncomprehensible for me and I'm afraid for most people too :) And yes, I meant the current estimate, as frequentist get the parameter estimate as given and data as random, the opposite of bayesian. $\endgroup$ – Curious Feb 17 '12 at 16:12
  • 3
    $\begingroup$ I think that the main issue in your definition Curious is, "...estimated value of a parameter would fall within the interval." It's not an estimated parameter but an unknown fixed parameter; and it doesn't fall within the interval, rather the interval moves around and 95% of the time captures the parameter. $\endgroup$ – John Feb 10 '14 at 21:23

Here's the most degenerate example. If I want to build a 95% confidence interval $I$ for a real-valued parameter $\mu$, I can use the following distribution:

$$ \mathbb P(I = (-\infty, +\infty)) = 0.95 \\ \mathbb P(I = \emptyset) = 0.05 $$

(Some definitions of confidence interval may not technically include infinite or empty intervals, but this doesn't affect the example). 95% of confidence intervals drawn from this distribution will contain $\mu$. But if I show you any particular confidence interval drawn from this distribution, your probability that it contains $\mu$ shouldn't be 0.95. It should be 1 for $(-\infty, +\infty)$ and 0 for $\emptyset$.


A "confidence interval" is a specific case of the broader concept of a "confidence set", which may or may not be a single connected interval. The broader concept can be conceived mathematically as follows. Suppose we have an observable data vector $\mathbf{X}_n \equiv (X_1,...,X_n)$ from a distribution with unknown parameter $\theta \in \Theta$. Then a confidence set for the parameter $\theta$ is created from a set function $\mathcal{S}$ that satisfies the following conditional probability requirement:$^\dagger$

$$1-\alpha = \mathbb{P}(\theta \in \mathcal{S}(\mathbf{X}_n, \alpha)|\theta) \quad \quad \quad \text{for all } \theta \in \Theta \text{ and } 0 \leqslant \alpha \leqslant 1.$$

Note that if $\theta$ is conceived as a random variable then this requirement also implies the following weaker property pertaining to the marginal probability of inclusion:

$$1-\alpha = \mathbb{P}(\theta \in \mathcal{S}(\mathbf{X}_n, \alpha)) \quad \quad \quad \text{for all } 0 \leqslant \alpha \leqslant 1. \quad \quad \quad \quad$$

Now, given a set function that comports to the above conditional probability requirement, for a given value $0 \leqslant \alpha \leqslant 1$, the confidence set for the data $\mathbf{x}_n$ (with confidence level $1-\alpha$) is the fixed set $\mathcal{S}(\mathbf{x}_n, \alpha)$. In the case where this is a single connected interval, we call it a confidence interval.

As can be seen, the confidence set is a fixed set determined by the observed data. As such, it is not possible to make any non-degenerate probability statement about its coverage of a fixed parameter. However, if we treat the data as random, we can see that the random confidence set will contain the conditioning value of the parameter $\theta$ with probability equal to the confidence level. This holds regardless of the conditioning value, and so it also holds as a marginal property if $\theta$ is a random variable. As discussed in a related answer, this is an extremely useful and robust property.

$^\dagger$ One slight complication here is that we sometimes form confidence intervals that only satisfy this probability requirement under some approximating assumption (e.g., distributional approximation using the central limit theorem). In these cases the probability requirement is satisfied exactly under some simplifying assumption and it holds approximately in broader cases.


Not the answer you're looking for? Browse other questions tagged or ask your own question.