# Clarification on interpreting confidence intervals?

My current understanding of the notion "confidence interval with confidence level $1 - \alpha$" is that if we tried to calculate the confidence interval many times (each time with a fresh sample), it would contain the correct parameter $1 - \alpha$ of the time.

Though I realize that this is not the same as "probability that the true parameter lies in this interval", there's something I want to clarify.

[Major Update]

Before we calculate a 95% confidence interval, there is a 95% probability that the interval we calculate will cover the true parameter. After we've calculated the confidence interval and obtained a particular interval $[a,b]$, we can no longer say this. We can't even make some sort of non-frequentist argument that we're 95% sure the true parameter will lie in $[a,b]$; for if we could, it would contradict counterexamples such as this one: What, precisely, is a confidence interval?

I don't want to make this a debate about the philosophy of probability; instead, I'm looking for a precise, mathematical explanation of the how and why seeing the particular interval $[a,b]$ changes (or doesn't change) the 95% probability we had before seeing that interval. If you argue that "after seeing the interval, the notion of probability no longer makes sense", then fine, let's work in an interpretation of probability in which it does make sense.

More precisely:

Suppose we program a computer to calculate a 95% confidence interval. The computer does some number crunching, calculates an interval, and refuses to show me the interval until I enter a password. Before I've entered the password and seen the interval (but after the computer has already calculated it), what's the probability that the interval will contain the true parameter? It's 95%, and this part is not up for debate: this is the interpretation of probability that I'm interested in for this particular question (I realize there are major philosophical issues that I'm suppressing, and this is intentional).

But as soon as I type in the password and make the computer show me the interval it calculated, the probability (that the interval contains the true parameter) could change. Any claim that this probability never changes would contradict the counterexample above. In this counterexample, the probability could change from 50% to 100%, but...

• Are there any examples where the probability changes to something other than 100% or 0% (EDIT: and if so, what are they)?

• Are there any examples where the probability doesn't change after seeing the particular interval $[a,b]$ (i.e. the probability that the true parameter lies in $[a,b]$ is still 95%)?

• How (and why) does the probability change in general after seeing the computer spit out $[a,b]$?

• This may provide some interesting points: en.wikipedia.org/wiki/Credible_interval – nico Jun 6 '11 at 5:16
• Your assumtions that P(E|C)=1 and P(E|C')=0 are unjustified. Why do you state that if the actual interval does not contain the true parameter value this latter one is surely outside it? – glassy Jun 6 '11 at 13:13
• I'm not sure what you mean by "actual interval" or "latter one". Can you please clarify? – Elliott Jun 6 '11 at 18:33
• @nico Thanks for the link. The intent of my original question was "my argument seems to show that a confidence interval can be interpreted as a Bayesian credible interval, but this is not the case, so what's wrong with my reasoning". But I have to admit, I'm not happy with the notion "the probability that the interval contains the [unknown] true parameter is either 0 or 1". To me, that's like saying "the probability that the coin landed heads, after I've flipped it but before I've looked at it, is either 0 or 1"; I don't see why it's not 1/2. – Elliott Jun 6 '11 at 18:38
• @Elliot: Schrödinger's cat comes to mind :) I'm not sufficiently expert to give you a proper explanation, but I would love to see an answer to this. PS: and let's not forget that the coin can also fall on the edge! – nico Jun 7 '11 at 11:06

I think the fundamental problem is that frequentist statistics can only assign a probability to something that can have a long run frequency. Whether the true value of a parameter lies in a particular interval or not doesn't have a long run frequency, becuase we can only perform the experiment once, so you can't assign a frequentist probability to it. The problem arises from the definition of a probability. If you change the definition of a probability to a Bayesian one, then the problem instantly dissapears as you are no longer tied to discussion of long run frequencies.

See my (rather tounge in cheek) answer to a related question here:

"A Frequentist is someone that believes probabilies represent long run frequencies with which events ocurr; if needs be, he will invent a fictitious population from which your particular situation could be considered a random sample so that he can meaningfully talk about long run frequencies. If you ask him a question about a particular situation, he will not give a direct answer, but instead make a statement about this (possibly imaginary) population."

In the case of a confidence interval, the question we normally would like to ask (unless we have a problem in quality control for example) is "given this sample of data, return the smallest interval that contains the true value of the parameter with probability X". However a frequentist can't do this as the experiment is only performed once and so there are no long run frequencies that can be used to assign a probability. So instead the frequentist has to invent a population of experiments (that you didn't perform) from which the experiment you did perform can be considered a random sample. The frequentist then gives you an indirect answer about that fictitious population of experiments, rather than a direct answer to the question you really wanted to ask about a particular experiment.

Essentially it is a problem of language, the frequentist definition of a popuation simply doesn't allow discussion of the probability of the true value of a parameter lying in a particular interval. That doesn't mean frequentist statistics are bad, or not useful, but it is important to know the limitations.

Regarding the major update

I am not sure we can say that "Before we calculate a 95% confidence interval, there is a 95% probability that the interval we calculate will cover the true parameter." within a frequentist framework. There is an implicit inference here that the long run frequency with which the true value of the parameter lies in confidence intervals constructed by some particular method is also the probability that that the true value of the parameter will lie in the confidence interval for the particular sample of data we are going to use. This is a perfectly reasonable inference, but it is a Bayesian inference, not a frequentist one, as the probability that the true value of the parameter lies in the confidence interval that we construct for a particular sample of data has no long run freqency, as we only have one sample of data. This is exactly the danger of frequentist statistics, common sense reasoning about probability is generally Bayesian, in that it is about the degree of plausibility of a proposition.

We can however "make some sort of non-frequentist argument that we're 95% sure the true parameter will lie in [a,b]", that is exactly what a Bayesian credible interval is, and for many problems the Bayesian credible interval exactly coincides with the frequentist confidence interval.

"I don't want to make this a debate about the philosophy of probability", sadly this is unavoidable, the reason you can't assign a frequentist probability to whether the true value of the statistic lies in the confidence interval is a direct consequence of the frequentist philosophy of probability. Frequentists can only assign probabilities to things that can have long run frequencies, as that is how frequentists define probability in their philosophy. That doesn't make frequentist philosophy wrong, but it is important to understand the bounds imposed by the definition of a probability.

"Before I've entered the password and seen the interval (but after the computer has already calculated it), what's the probability that the interval will contain the true parameter? It's 95%, and this part is not up for debate:" This is incorrect, or at least in making such a statement, you have departed from the framework of frequentist statistics and have made a Bayesian inference involving a degree of plausibility in the truth of a statement, rather than a long run frequency. However, as I have said earlier, it is a perfectly reasonable and natural inference.

Nothing has changed before or after entering the password, because niether event can be assigned a frequentist probability. Frequentist statistics can be rather counter-intuitive as we often want to ask questions about degrees of plausibility of statements regarding particular events, but this lies outside the remit of frequentist statistics, and this is the origin of most misinterpretations of frequentist procedures.

• Yes, that is very much the point I am making, the second statement is not actually a statement about this particular coin. It is a statement about an imaginary population of coins that most people incorrectly intepret as a statement about our particular coin. However in making that leap we are applying Bayesian intuition about probability, and ignoring what a confidence interval actually is. There is no problem in assigning a probability to the state of the coin, provided we move away from a frequentist definition of probability. – Dikran Marsupial Jun 8 '11 at 19:29
• To clarify, to say that "imagine many separate instances of the beginning of time; you'd expect about half of those to produce heads" is perfectly correct frequentist reasoning. However to go from there to "therefore the probability that this particlar coin is also 0.5" is not as a probability is applied to something that doesn't have a long run frequency as it can only happen once and once only. It is perfectly sound Bayesian reasoning though, as a Bayesian probability is a statement about the plausibility of a proposition (which can be based on a long run frequency within a population). – Dikran Marsupial Jun 8 '11 at 19:41
• His answer to both questions would actually be a statement about the proportion of an imaginary population of coins that would land heads. But it is likely that this would not be made explicit, as people generally like to be helpful (indirect answers are not generally that helpfull), and also frequentist statistics is rather counter-intuitive and the frequentist is likely to skirt around this point to avoid confusion. If pinned down to make a probabilistic statment about a particular flip, a good frequentist would simply refuse to answer - it lies outside the bounds of frequentist stats. – Dikran Marsupial Jun 8 '11 at 19:56
• Essentially the frequentist wouldn't actually answer your question, he'd make a statement about a population of coin flips and leave you to infer from that that the probability of a head on that particular flip was the same as the proportion in the implicit population. But that would be your Bayesian inference, not his. – Dikran Marsupial Jun 8 '11 at 20:03
• @Aaron Sure, you could say "the probability is either 0 or 1" for many things, but that answer buys us absolutely nothing (for example, if we want to answer a concrete question about how much we should bet on a game or whether or not we should launch a space shuttle). Besides, the things that "could happen" are: (1) the coin landed heads and you covered it, (2) the coin landed tails and you covered it; in an "imaginary population" of many "flipping and covering" trials, about 50% result in you seeing heads. – Elliott Jun 8 '11 at 22:28

Major update, major new answer. Let me try to clearly address this point, because it's where the problem lies:

"If you argue that "after seeing the interval, the notion of probability no longer makes sense", then fine, let's work in an interpretation of probability in which it does make sense."

The rules of probability don't change but your model for the universe does. Are you willing to quantify your prior beliefs about a parameter using a probability distribution? Is updating that probability distribution after seeing the data a reasonable thing to do? If you think so then you can make statements like $P(\theta\in [L(X), U(X)]| X=x)$. My prior distribution can represent my uncertainty about the true state of nature, not just randomness as it is commonly understood - that is, if I assign a prior distribution to the number of red balls in an urn that doesn't mean I think the number of red balls is random. It's fixed, but I'm uncertain about it.

Several people including I have said this, but if you aren't willing to call $\theta$ a random variable then the statement $P(\theta\in [L(X), U(X)]| X=x)$ isn't meaningful. If I'm a frequentist, I'm treating $\theta$ as a fixed quantity AND I can't ascribe a probability distribution to it. Why? Because it's fixed, and my interpretation of probability is in terms of long-run frequencies. The number of red balls in the urn doesn't ever change. $\theta$ is what $\theta$ is. If I pull out a few balls then I have a random sample. I can ask what would happen if I took a bunch of random samples - that is to say, I can talk about $P(\theta\in [L(X), U(X)])$ because the interval depends on the sample, which is (wait for it!) random.

But you don't want that. You want $P(\theta\in [L(X), U(X)]| X=x)$ - what's the probability that this interval I constructed with my observed (and now fixed) sample contains the parameter. However, once you've conditioned on $X=x$ then to me, a frequentist, there is nothing random left and the statement $P(\theta\in [L(X), U(X)]| X=x)$ doesn't make sense in any meaningful way.

The only principled way (IMO) to make a statement about $P(\theta\in [L(X), U(X)]| X=x)$ is to quantify our uncertainty about a parameter with a (prior) probability distribution and update that distribution with new information via Bayes Theorem. Every other approach I have seen is a lackluster approximation to Bayes. You certainly can't do it from a frequentist perspective.

That isn't to say that you can't evaluate traditional frequentist procedures from a Bayesian perspective (often confidence intervals are just credible intervals under uniform priors, for example) or that evaluating Bayesian estimators/credible intervals from a frequentist perspective isn't valuable (I think it can be). It isn't to say that classical/frequentist statistics is useless, because it isn't. It is what it is, and we shouldn't try to make it more.

Do you think it's reasonable to give a parameter a prior distribution to represent your beliefs about the universe? It sounds like it from your comments that you do; in my experience most people would agree (that's the little half-joke I made in my comment to @G. Jay Kerns's answer). If so, the Bayesian paradigm provides a logical, coherent way to make statements about $P(\theta\in [L(X), U(X)]| X=x)$. The frequentist approach simply doesn't.

• (+1) very well done, again, and spot on dead-center. – user1108 Jun 9 '11 at 12:27
• +1 Same comment as above (see G. Jay Kerns' answer); this was really helpful. – Elliott Jun 12 '11 at 9:00
• Bounty schmounty :) I'm glad you found it helpful. – JMS Jun 12 '11 at 17:05

OK, now you're talking! I've voted to delete my previous answer because it doesn't make sense with this major-updated question.

In this new, updated question, with a computer that calculates 95% confidence intervals, under the orthodox frequentist interpretation, here are the answers to your questions:

1. No.
2. No.
3. Once the interval is observed, it is not random any more, and does not change. (Maybe the interval was $[1,3]$.) But $\theta$ doesn't change, either, and has never changed. (Maybe it is $\theta = 7$.) The probability changes from 95% to 0% because 95% of the intervals the computer calculates cover 7, but 100% of the intervals $[1,3]$ do NOT cover 7.

(By the way, in the real world, the experimenter never knows that $\theta = 7$, which means the experimenter can never know whether the true probability $[1,3]$ covers $\theta$ is zero or one. (S)he only can say that it must be one or the other.) That, plus the experimenter can say that 95% of the computer's intervals cover $\theta$, but we knew that already.

The spirit of your question keeps hinting back to the observer's knowledge, and how that relates to where $\theta$ lies. That (presumably) is why you were talking about the password, about the computer calculating the interval without your seeing it yet, etc. I've seen in your comments to answers that it seems unsatisfactory/unseemly to be obliged to commit to 0 or 1, after all, why couldn't we believe it is 87%, or $15/16$, or even 99%??? But that is exactly the power - and simultaneously the Achilles' heel - of the frequentist framework: the subjective knowledge/belief of the observer is irrelevant. All that matters is a long-run relative frequency. Nothing more, nothing less.

As a final BTW: if you change your interpretation of probability (which you intentially have elected not to do for this question), then the new answers are:

1. Yes.
2. Yes.
3. The probability changes because probability = subjective knowledge, or degree of belief, and the knowledge of the observer changed. We represent knowledge with prior/posterior distributions, and as new information becomes available, the former morphs into the latter (via Bayes' Rule).

(But for full disclosure, the setup you describe doesn't match the subjective interpretation very well. For instance, we usually have a 95% prior credible interval before even turning on the computer, then we fire it up and employ the computer to give us a 95% posterior credible interval which is usually considerably skinnier than the prior one.)

• Again! :) Well done. – JMS Jun 9 '11 at 3:39
• I should just point out that the Bayesian intepretation of probability is not necessarily subjective, so that is not really a strength of the frequentist approach. In the case of the objective Bayesian approach to the "probability of a coin landing heads up problem", using an uninformative prior involves no subjectivity at all. The real strength of the frequentist approach lies in problems such as quality control, where is is natural to talk of repeated trials and long run frequencies. It only has difficulties when you pose questions about particular events. – Dikran Marsupial Jun 9 '11 at 9:10
• @JMS, thanks. @Dikran, it's difficult to talk about it with 544 characters in a small box on computer screen. Briefly: I agree with you that the word "Bayesian" isn't synonymous with "subjective". And there's no point trying to nail down where the real strength of either approach really lies. Bottom line: we can all agree on a long-run relative frequency, but more often than not, your posterior will be different from mine. – user1108 Jun 9 '11 at 12:13
• @Dikran Marsupial You make a good point. I'd only add that once we move past toy problems into actual applied modeling it is often the case that the truly important subjectivity comes in through how we specify the likelihood, not necessarily the prior distributions themselves (Are those observations really exchangeable? Gaussian? etc). In that way subjectivity is built into a huge portion of model-based statistics, Bayesian and frequentist alike. – JMS Jun 9 '11 at 14:15
• +1 Thanks for the beautiful answer. This definitely deserves a bounty, but to avoid being political, I ended up going by upvotes. – Elliott Jun 12 '11 at 8:59

I'll throw in my two cents (maybe redigesting some of the former answers). To a frequentist, the confidence interval itself is in essence a two-dimensional random variable: if you would redo the experiment a gazillion times, the confidence interval you would estimate (i.e.: calculate from your newly found data each time) would differ each time. As such, the two boundaries of the interval are random variables.

A 95% CI, then, means nothing more than the assurance (given all your assumptions leading to this CI are correct) that this set of random variables will contain the true value (a very frequentist expression) in 95% of the cases.

You can easily calculate the confidence interval for the mean of 100 draws from a standard normal distribution. Then, if you draw 10000 times 100 values from that standard normal distribution, and each time calculate the confidence interval for the mean, you will indeed see that 0 is in there about 9500 times.

The fact that you have created a confidence interval just once (from your actual data) does indeed reduce the probability of the true value being in that interval to either 0 or 1, but it doesn't change the probability of the confidence interval as a random variable to contain the true value.

So, bottom line: the probability of any (i.e. on average) 95% confidence interval containing the true value (95%) doesn't change, and neither does the probability of a particular interval (CI or whatever) for containing the true value (0 or 1). The probability of the interval the computer knows but you don't is actually 0 or 1 (because it is a particular interval), but since you don't know it (and, in a frequentist fashion, are unable to recalculate this same interval infinitely many times again from the same data), all you have to go for is the probability of any interval.

• Funny side note: this site's spelling checker finds the word frequentist to be worthy of curly underlining. Is this site secretly mastered by bayesians? Oh, I guess it's not, since bayesians have their own curly underlining :-) – Nick Sabbe Jun 9 '11 at 22:01

The reason that the confidence interval doesn't specify "the probability that the true parameter lies in the interval" is because once the interval is specified, the paramater either lies in it or it doesn't. However, for a 95% confidence interval for example, you have a 95% chance of creating a confidence interval that does contain the value. This is a pretty difficult concept to grasp, so I may not be articulating it well. See http://frank.itlab.us/datamodel/node39.html for further clarification.

• Suppose you program a computer to calculate a confidence interval, but you don't look at the output. Before you've seen the output, you know that there's a 95% chance that the interval contained the right parameter (just like before you've seen the result of a coin flip, you know there's a 50% chance of heads). In what way does subsequently looking at the output change this probability, given that you don't know the right parameter in the first place (I agree that looking at the result of a coin flip changes the probability of heads from 50% to either 1 or 0)? – Elliott Jun 6 '11 at 4:22
• Furthermore, while I agree that there's a distinction, I'm wondering what's wrong with my above "argument" showing that they're the same thing. – Elliott Jun 6 '11 at 4:23
• @Elliott Your questions seem to be the same as this analogy: you flip a fair coin. Ergo, the probability of heads is 50%. Now you look at the coin and it is heads. In what way does this change the probability of heads? The answer is that it doesn't, because the probability refers to the coin-flipping procedure, not to the outcome. It seems to me that the counterexample you refer to works in a similar fashion: the procedure may have a 50% chance of covering the parameter, but after the fact it is has been possible to verify that the parameter indeed is covered. So what? – whuber Jun 6 '11 at 16:09
• I'm not talking about changing the probability that a fair coin will be heads; instead, I'm talking about changing the probability that this particular coin will be heads. After I've flipped it and before I've looked at it, I would argue that the probability in question is 50% because about half of such cases involve a coin with heads up. On the other hand, after I've looked at it and seen heads, 100% of such cases involve a coin with heads up (the cases with tails up were eliminated when I looked at the coin and didn't see heads). – Elliott Jun 6 '11 at 18:29
• I agree that after the fact, it might be possible to verify that the parameter has been covered. My answer to "so what?" is "so my above argument (in the original question) must be wrong, and I'm wondering what's wrong with it". – Elliott Jun 6 '11 at 18:32

I don't think a frequentist can say there is any probability of the true (population) value of a statistic lying in the confidence interval for a particular sample. It either is, or it isn't, but there is no long run frequency for a particular event, just the population of events that you would get by repeated performance of a statistical procedure. This is why we have to stick with statements such as "95% of confidence intervals so constructed will contain the true value of the statistic", but not "there is a p% probability that the true value lies in the confidence interval computed for this particular sample". This is true for any value of p, it simply isn't possible withing the frequentist definition of what a probability actually is. A Bayesian can make such a statement using a credible interval though.

The way you pose the problem is a little muddled. Take this statement: Let $E$ be the event that the true parameter falls in the interval $[a,b]$. This statement is meaningless from a frequentist perspective; the parameter is the parameter and it doesn't fall anywhere, it just is. P(E) is meaningless, P(E|C) is meaningless and this is why your example falls apart. The problem isn't conditioning on a set of measure zero either; the problem is that you're trying to make probability statements about something that isn't a random variable.

A frequentist would say something like: Let $\tilde E$ be the event that the interval $(L(X), U(X))$ contains the true parameter. This is something a frequentist can assign a probability to.

Edit: @G. Jay Kerns makes the argument better than me, and types faster, so probably just move along :)

• Thanks, I see why a frequentist refuses to answer the question "what's the probability that the true parameter falls in a given interval" (even though I think this is a useful thing to talk about, especially if one doesn't know the true parameter). But why am I not conditioning on a null set if I condition on the event that my calculation returns precisely the interval $[a,b]$? – Elliott Jun 8 '11 at 22:46
• A (principled/pedantic) frequentist doesn't refuse to answer, he can't answer. The parameter isn't random, the interval is. What's in front of the $|$ when you condition on observing a particular interval? For a frequentist, once you've observed the sample and constructed your interval the interval, nothing is random any more. – JMS Jun 8 '11 at 22:59
• I don't know, yours is spot on. :-) – user1108 Jun 9 '11 at 0:39

In frequentist statistics, the event $E$ is fixed -- the parameter either lies in $[a, b]$ or it doesn't. Thus, $E$ is independent of $C$ and $C'$ and so both $P(E|C) = P(E)$ and $P(E|C') = P(E)$.

(In your argument, you seem to think that $P(E|C) = 1$ and $P(E|C') = 0$, which is incorrect.)

• Considering that I defined C to be the event that this particular interval, [a, b], contains the true parameter, I'm not sure I agree that E and C / C' are independent: knowing that C occurred guarantees that E occurred. – Elliott Jun 8 '11 at 19:11
• But C is a random variable! You're kind of changing the definitions of all these events after everything has already occurred. Put another way, if you're defining C to be this particular event, then C is no longer a confidence interval. – raegtin Jun 8 '11 at 19:16
• The problem is that if C is the event that the interval contains the true parameter in this particular run of the experiment, it doesn't have a long run frequency (that particular run can only happen once), and thus you can't assign a frequentist probability to it. This is why the definition of a frequentist confidence interval is in terms of a population of replicated experiments. You appear to be applying Bayesian reasoning to a frequentist setting, and there is a mis-match of definitions of probability. – Dikran Marsupial Jun 8 '11 at 19:19
• Here's another way to look at it. What you seem to be doing is the following: run a calculation to get a confidence interval [a, b]. Define C to be the event that this particular confidence interval [a, b] contains the true parameter. Also define E to be the event that this particular interval [a, b] contains the true parameter. Thus, E and C are the same event! – raegtin Jun 8 '11 at 19:25
• That's what you're actually doing. It seems like you think you're doing the following (which you're not): run calculation #1 to get an interval [a, b]. Define E to be the event that this particular interval [a, b] contains the true parameter. Next, forget about calculation #1, and define C to be the event that any other calculated interval [a', b'] contains the true parameter. In this case, E and C are independent. – raegtin Jun 8 '11 at 19:28

There are so many long explanations here that I don't have time to read them. But I think the answer to the basic question can be short and sweet. It is the difference between a probability that is unconditional on the data. The probability of 1-alpha before collecting the dats is the probability that the well-defined procedure will include the parameter. After you have collected the data and know the specific interval that you have generated the interval is fixed and so since the parameter is a constant this conditional probability is either 0 or 1. But since we don't know the actual value of the parameter even after collecting the data we don't know which value it is.

Extension of the post by Michael Chernick copied form comments:

there is a pathological exception to this which can be called perfect estimation. Suppose we have a first order autoregressive process given by X(n)=pX(n-1) + en. It is stationary so we know p is not 1 or -1 and is < 1 in absolute value. Now the en are independent identically distributed with a mixed distribution there is a positive probability q that en= 0

There is a pathological exception to this which can be called perfect estimation. Suppose we have a first order autoregressive process given by X(n)=pX(n-1) + en. It is stationary so we know p is not 1 or -1 and is < 1 in absolute value.

Now the en are independent identically distributed with a mixed distribution there is a positive probability q that en=0 and with probability 1-q it has an absolutely continuous distribution (say that the density is non zero in an interval bounded away from 0. Then collect data from the time series sequentially and for each successive pair of values estimate p by X(i)/X(i-1). Now when ei = 0 the ratio will equal p exactly.

Because q is greater than 0 eventually the ratio will repeat a value and that value has to be the exact value of the parameter p because if it is not the value of ei which is not 0 will repeat with probability 0 and ei/x(i-1) will not repeat.

So the sequential stopping rule is to sample until the ratio repeats exactly then use the repeated value as the estimate of p. Since it is p exactly any interval you construct that is centered at this estimate has probability 1 of including the true parameter. Although this is a pathological example that is not practical there do exist stationary stochastic processes with the properties that we require for the error distribution

• have you considered appending the example described over the course of these several comments onto your answer, instead? – Macro May 5 '12 at 5:04
• @Michael I'll second Macro's comment. Please note that comments are generally thought of as a way to interact with other users (e.g., when requesting clarifications, etc.), and in any case are sometimes viewed as 'third-class citizens in Stack Exchange'. However, following our most recent exchange, I'll let you decide on how to proceed with this series of comments. This remark applies to another series of comments found here. – chl May 5 '12 at 19:03
• I don't put comments in answers because there seems to be a policy to downvote answers that have a lot of discussion in them particualrly when someone judges that the answer doesn't really answer the question. So answers are to give answers and comments go under comments. My comments tend to be over the character limit so I use several. – Michael R. Chernick May 6 '12 at 4:06
• @MichaelChernick There is no such policy, thus I've incorporated your comments into the post. – user88 May 8 '12 at 20:50
• @MichaelChernick, I've been posting on this site pretty regularly for around a year and I've never heard anyone suggesting that the moderators were oppressive or that the rules of the site were confusing. The issues that you've run into re: your posts are things that are explicitly discussed in the FAQ. – Macro May 9 '12 at 3:27

Two observations about the many questions and responses that may help still.

Part of the confusion comes from glossing over some deeper math of probability theory, which, by the way, was not on a firm mathematical footing until about the 1940s. It gets into what constitutes sample spaces, probability spaces, etc.

First, you had stated that after a coin flip we know that there is 0% probability it did not come up tails if it came up heads. At that point it doesn't make sense to talk about probability; what happened happened, and we know it. Probability is about the unknown in the future, not the known in the present.

As a small corollary to that about what zero probability really means, consider this: we assume a fair count has a probability of 0.5 of coming up heads, and 0.5 of coming up tails. This means it has a 100% chance of coming up either heads or tails, since those outcomes are MECE (mutually exclusive and completely exhaustive). It has a zero percent change, however, of comping up heads and tails: Our notion of 'heads' and 'tails' are that they are mutually exclusive. Thus, this has zero percent chance because it is impossible in the way we think of (or define) 'tossing a coin'. And it is impossible before and after the toss.

As a further corollary to this, anything that is not, by definition, impossible is possible. In the real world, I hate when lawyers ask "isn't it possible you signed this document and forgot about it?" because the answer is always 'yes' by the nature of the question. For that matter, the answer is also 'yes' to the question "isn't it possible you were transported through dematerialization to planet Remulak 4 and forced to do something then transported back with no memory of it?". The likelihood may be very low -but what is not impossible is possible. In our regular concept of probability, when we talk about flipping a coin, it may come up heads; it may come up tails; and it may even stand on-end or (somehow, such as if we were snuck into a spacecraft while drugged and taken into orbit) float in the air forever. But, before or after the toss, it has zero probability of coming up heads and tails at the same time: they are mutually exclusive outcomes in the sample space of the experiment(look up 'probability sample spaces' and 'sigma-algebras').

Second, on all this Bayesian/Frequentist philosophy on confidence intervals, it is true it relates to frequencies if one is acting as a frequentist. So, when we say the confidence interval for a sampled and estimated mean is 95%, we are not saying that we are 95% certain the 'real' value lies between the bounds. We are saying that, if we could repeat this experiment over-and-over, 95% of the time we would find that the mean was, indeed, between the bounds. When we do it with one run, however, we are taking a mental shortcut and saying 'we are 95% certain we are right'.

FInally, don't forget what the standard setup is on a hypothesis test based on an experiment. If we want to know if a plant growth hormone makes plants grow faster, maybe we first determine the average size of a tomato after 6 months of growth. Then we repeat, but with the hormone, and get the average size. Our null hypothesis is 'the hormone didn't work', and we test that. But, if the tested plants are, on average larger, with 99% confidence, that means 'there will always be random variation due to the plants and how accurately we weigh, but the amount of randomness that would explain this would occur less than one time in a hundred."

The issue can be characterized as a confusion of prior and posterior probability or maybe as the dissatisfaction of not knowing the joint distribution of certain random variables.

Conditioning

As an introductory example, we consider a model for the experiment of drawing, without replacement, two balls from an urn with $$n$$ balls numbered from $$1$$ to $$n$$. The typical way to model this experiment is with two random variables $$X$$ and $$Y$$, where $$X$$ is the number of the first ball and $$Y$$ is the number of the second ball, and with the joint distribution $$P(X=x \land Y=y) = 1/(n(n-1))$$ for all $$x,y \in N := \{1,\dots,n\}$$ with $$x \neq y$$. This way, all possible outcomes have the same, positive probability, and the impossible outcomes (e.g., drawing the same ball twice) have zero probability. It follows $$P(X=x)=1/n$$ and $$P(Y=x)=1/n$$ for all $$x \in N$$.

Let the experiment be conducted and the second ball revealed to us, while the first ball is kept secret. Denote $$t$$ the number of the second ball. Then, still, $$P(X=x)=1/n$$ for all $$x \in N$$. However, for each $$x \in N$$, our degree of belief that the event $$X=x$$ happened, should now be $$P(X=x \vert Y=t) = P(X=x \land Y=t) / P(Y=t)$$, which in case of $$x \neq t$$ is $$1/(n-1)$$, and in case of $$x = t$$, it is $$0$$. This is the probability of $$X=x$$ conditioned on the information that $$Y=t$$ happened, also called the posterior probability of $$X=x$$, meaning, the updated probability of $$X=x$$ after we obtained the evidence that $$Y=t$$ happened. It is still $$P(X=x)=P(Y=x)=1/n$$ for all $$x \in N$$, those are the prior probabilities.

Not conditioning on evidence means ignoring evidence. However, we can only condition on what is expressible in the probabilistic model. In our example with the two balls from the urn, we cannot condition on the weather or on how we feel today. In case that we have reason to believe that such is evidence relevant to the experiment, we must change our model first in order to allow us to express this evidence as formal events.

Let $$C$$ be the indicator random variable that says if the first ball has a lower number than the second ball, that is, $$C = 1 \Longleftrightarrow X < Y$$. Then $$P(C=1) = 1/2$$. Let again $$t$$ be the number of the second ball, which is revealed to us, but the number of the first ball is secret. Then it is easy to see that $$P(C=1 \vert Y=t) = (t-1) / (n-1)$$. In particular $$P(C=1 \vert Y=1) = 0$$, which in our model means that $$C=1$$ has certainly not happened. Moreover, $$P(C=1 \vert Y=n) = 1$$, which in our model means that $$C=1$$ has certainly happened. It is still $$P(C=1) = 1/2$$.

Confidence Interval

Let $$X = (X_1, \dots, X_n)$$ be a vector of $$n$$ i.i.d random variables. Let $$(l,u)$$ be a confidence interval estimator (CIE) with confidence level $$\gamma$$ for a real parameter of the distribution of the random variables in $$X$$, that is, $$l$$ and $$u$$ are real-valued functions with domain $$\mathbb{R}^n$$, such that if $$\theta \in \mathbb{R}$$ is the true value of the parameter, then $$P(l(X) \leq \theta \leq u(X)) \geq \gamma$$.

Let $$C$$ be the indicator random variable that says if $$(l,u)$$ determined the correct parameter, that is, $$C = 1 \Longleftrightarrow l(X) \leq \theta \leq u(X)$$. Then $$P(C=1) \geq \gamma$$.

Let us collect data so that we have values $$x = (x_1,\dots,x_n) \in \mathbb{R}^n$$, where $$x_i$$ is the realization of $$X_i$$ for all $$i$$. Then our degree of belief that the event $$C=1$$ happened should be $$\delta := P(C=1 \vert X = x)$$. In general, we cannot compute this conditional probability, but we know that it is either $$0$$ or $$1$$, since $$(C = 1 \land X = x) \Longleftrightarrow ((l(x) \leq \theta \leq u(x)) \land X = x)$$. If $$l(x) \leq \theta \leq u(x)$$ is false, then the latter statement is false, and thus $$\delta=0$$. If $$l(x) \leq \theta \leq u(x)$$ is true, then the latter statement is equivalent to $$X=x$$, and thus $$\delta=1$$. If we only know the values $$l(x)$$ and $$u(x)$$ and not the data $$x$$, we can still argue in a similar way that $$\delta \in \{0,1\}$$.

It is still $$P(C=1) \geq \gamma$$. If, for our degree of belief that $$C=1$$ happened, we like this prior probability more, then we must ignore $$x$$, and this also means ignoring the confidence interval $$[l(x),u(x)]$$. Saying that $$[l(x),u(x)]$$ contained $$\theta$$ with probability at least $$\gamma$$, would mean acknowledging this evidence and at the same time ignoring it.

Learning More, Knowing Less

What makes this situation so difficult to grasp may be the fact that we cannot compute the conditional probability $$\delta$$. But this is not particular to the CIE situation, rather it may occur whenever we have insufficient information about the joint distribution of random variables. As an example, let $$X$$ and $$Y$$ be discrete random variables and let their marginal distributions be given, that is, for each $$x \in \mathbb{R}$$, we know $$P(X=x)$$ and $$P(Y=x)$$. This does not give us their joint distribution, that is, we do not know $$P(X=x \land Y=y)$$ for any $$x,y \in \mathbb{R}$$. Assume that a result of this experiment should be reported as the value of the random vector $$(X,Y)$$, that is, results should be reported as pairs of real numbers.

Let the underlying experiment be conducted, and assume that we learn that $$Y=7$$ happened, while the value for $$X$$ is still unknown to us. This does not change $$P(X=x)$$ for any $$x$$. However, it would be problematic to say that the result of the experiment was of the form $$(x,7)$$, where $$x \in \mathbb{R}$$, and that the probability for each particular real number $$x$$ for being the first component of this pair was $$P(X=x)$$. It is problematic since in this way, we would acknowledge the evidence $$Y=7$$ and at the same time ignore it. We acknowledge the evidence $$Y=7$$ by reporting the second component of the pair as being $$7$$. We ignore it by using the prior probability $$P(X=x)$$, where in fact our degree of belief for $$X=x$$ should now be $$P(X=x \vert Y=7) = P(X=x \land Y=7) / P(Y=7)$$, which unfortunately we cannot compute.

It may be unsatisfactory that in a sense, knowing more about $$Y$$ forces us to say less about $$X$$. But to the best of my knowledge this is how things are.

If I say the probability the Knicks scored between xbar - 2sd(x) and xbar + 2sd(x) is about .95 in some given game in the past, that is a reasonable statement given some particular distributional assumption about the distribution of basketball scores. If I gather data about the scores given some sample of games and calculate that interval, the probability that they scored in that interval on some given day in the past is clearly zero or one, and you can google the game result to find out. The only notion of it maintaining non-zero or one probability to the frequentist comes from repeated sampling, and the realization of interval estimation of a particular sample is the magic point where either it happened or it didn't given the interval estimate of that sample. It isn't the point where you type in the password, it is the point where you decided to take a single sample that you lose the continuity of possible probabilities.

This is what Dikran argues above, and I have voted up his answer. The point when repeated samples are out of the consideration is the point in the frequentist paradigm where the non-discrete probability becomes unobtainable, not when you type in the password as in your example above, or when you google the result in my example of the Knicks game, but the point when your number of samples =1.

Modeling

Correct procedure is: (1) model the situation as a probability space $\mathcal{S} = (\Omega,\Sigma,P)$; (2) define an event $E \in \Sigma$ of interest; (3) determine its probability $P(E)$. The event $E$ may be specified via random variables, that is, functions on $\mathcal{S}$ (measurable functions, that is, but let's not worry about this here). The space $\mathcal{S}$ may be given implicitly by one or more random variables and their joint distribution.

Step (1) may allow some leeway. The appropriateness of the modeling can sometimes be tested by comparing the probability of certain events with what we would expect intuitively. In particular, looking at certain marginal or conditional probabilities may help to get an idea how appropriate the modeling is.

Sometimes, modeling or a part of it has already been done and we can build on this. In statistics (at a certain point), we typically are already given real-valued random variables $X_1, \dots, X_n \sim \mathrm{Dist}(\theta)$ i.i.d with fixed but unknown ${\theta \in \mathbb{R}}$.

Confidence Interval Estimator

A confidence interval estimator (CIE) at the $\gamma$ confidence level is a pair of functions $L$ and $R$ with domain $\mathbb{R}^n$ such that $P(L(X) \leq \theta \leq R(X)) \geq \gamma$, writing $X = (X_1, \dots, X_n)$. I prefer the wording "confidence interval estimator" to underline that it is the functions and their functional properties that count; $L(X)$ and $R(X)$ are both functions on the implicitly given sample space, that is, they are random variables. Given an observation $x \in \mathbb{R}^n$, speaking of the "probability" of $L(x) \leq \theta \leq R(x)$ makes no sense since this is not an event since it does not contain any random variables.

Preferences

Suppose one may choose between a lottery ticket that has been drawn from a set of tickets where a $\gamma_1$ fraction consists of winning tickets, and one that has been drawn from a set where a $\gamma_2$ fraction consists of winning tickets, and suppose $\gamma_1 < \gamma_2$. Both tickets have already been drawn, but none of them revealed. Of course, all else being equal, we would prefer the second ticket, since it had a higher probability of being a winning ticket than the first one when they were drawn. A preference regarding different observations (the two tickets in this examples) based on the probabilistic properties of the random processes that generated the observations is fine. Note that we do not say that any of the tickets has a higher probability of being a winning ticket. If we ever say so, then with "probability" in a colloquial sense, which could mean anything, so it is best avoided here.

With CIEs of different confidence levels, all else is usually not equal, since higher confidence level will make the intervals delivered by the CIE tend to be wider. So we cannot even give a preference in this case; we cannot say that we generally prefer intervals computed with a CIE that has higher confidence level. But if all else was equal, we would prefer intervals produced by a CIE that has highest available confidence level. For example, if we were to choose between an interval that is the output of a CIE at the $0.95$ confidence level and an interval of the same length that has been drawn uniformly at random from the set of all intervals of this length, we would certainly prefer the former.

Example with a Simple Prior

Let us consider an example where the probabilistic modeling has been extended in order to make the parameter we are interested in a random variable. Suppose $\theta$ is a discrete random variable with $P(\theta=0) = P(\theta=1) = 1/2$ and that for each $\vartheta \in \mathbb{R}$, conditioned on the knowledge of $\theta = \vartheta$, we have $X_1, \dots, X_n \sim \mathcal{N}(\vartheta, 1)$ i.i.d. Let $L,R$ constitute a (classical) CIE for the mean of the normal distribution at the $\gamma$ confidence level, that is, for each $\vartheta \in \mathbb{R}$, we have $P(L(X) \leq \vartheta \leq R(X) \vert \theta = \vartheta) \geq \gamma$, which implies ${P(L(X) \leq \theta \leq R(X)) \geq \gamma}$.

Suppose we observe a concrete value $x \in \mathbb{R}^n$ of the $(X_1, \dots, X_n)$. Now, what is the probability of $\theta$ being located inside of the interval specified by $L(x)$ and $R(x)$, that is, what is $P(L(x) \leq \theta \leq R(x) \vert X = x)$? Denote $f_\mu$ the joint PDF of $n$ independent, normally distributed random variables with mean $\mu$ and standard deviation $\sigma=1$. A calculation using Bayes' rule and the law of total probability shows: $$P(L(x) \leq \theta \leq R(x) \vert X = x) = \begin{cases} \frac{f_0(x)}{f_0(x) + f_1(x)} & \text{if L(x) \leq 0 \leq R(x) < 1} \\ \frac{f_1(x)}{f_0(x) + f_1(x)} & \text{if 0 < L(x) \leq 1 \leq R(x)} \\ 1 & \text{if L(x) \leq 0 and 1 \leq R(x)} \\ 0 & \text{else} \end{cases}$$ Remarkably, this probability has nothing to do with the confidence level $\gamma$ at all! So even if the question for the probability of $\theta$ being contained in the output of the CIE makes sense, that is, if $L(X) \leq \theta \leq R(X)$ is an event in our probabilistic model, its probability in general is not $\gamma$, but can be something completely different.

In fact, once we have agreed on a prior (such as the simple discrete distribution of $\theta$ here) and we have an observation $x$, it may be more informative to condition on $x$ than looking at the output of a CIE. Precisely, for $\{\mu_0,\mu_1\} = \{0,1\}$ we have: $$P(\theta = \mu_0 \vert X=x) = \frac{f_{\mu_0}(x)}{f_{\mu_0}(x) + f_{\mu_1}(x)}$$

If we could say "the probability that the true parameter lies in this confidence interval" then we wouldn't take into account the size of the sample. No matter how large the sample is, as long as the the mean is the same, then the confidence interval would be equally wide. But when we say "if i repeat this 100 times, then I would expect that in 95 of the cases the true parameter will lie within the interval", we are taking into account the size of the sample size, and how sure our estimate is. The larger the sample size is, the less variance will the mean estimate have. So it wont vary that much, and when we are repeating the procedure 100 times, we doesn't need a large interval to make sure that in 95 of the cases the true parameter is in the interval.

• Keep in mind that confidence interval is a frequentist concept. – Michael R. Chernick Oct 21 '19 at 17:02