Clarification on interpreting confidence intervals?

Question

My current understanding of the notion "confidence interval with confidence level $1 - \alpha$" is that if we tried to calculate the confidence interval many times (each time with a fresh sample), it would contain the correct parameter $1 - \alpha$ of the time.

Though I realize that this is not the same as "probability that the true parameter lies in this interval", there's something I want to clarify.

[Major Update]

Before we calculate a 95% confidence interval, there is a 95% probability that the interval we calculate will cover the true parameter. After we've calculated the confidence interval and obtained a particular interval $[a,b]$, we can no longer say this. We can't even make some sort of non-frequentist argument that we're 95% sure the true parameter will lie in $[a,b]$; for if we could, it would contradict counterexamples such as this one: What, precisely, is a confidence interval?

I don't want to make this a debate about the philosophy of probability; instead, I'm looking for a precise, mathematical explanation of the how and why seeing the particular interval $[a,b]$ changes (or doesn't change) the 95% probability we had before seeing that interval. If you argue that "after seeing the interval, the notion of probability no longer makes sense", then fine, let's work in an interpretation of probability in which it does make sense.

More precisely:

Suppose we program a computer to calculate a 95% confidence interval. The computer does some number crunching, calculates an interval, and refuses to show me the interval until I enter a password. Before I've entered the password and seen the interval (but after the computer has already calculated it), what's the probability that the interval will contain the true parameter? It's 95%, and this part is not up for debate: this is the interpretation of probability that I'm interested in for this particular question (I realize there are major philosophical issues that I'm suppressing, and this is intentional).

But as soon as I type in the password and make the computer show me the interval it calculated, the probability (that the interval contains the true parameter) could change. Any claim that this probability never changes would contradict the counterexample above. In this counterexample, the probability could change from 50% to 100%, but...

• Are there any examples where the probability changes to something other than 100% or 0% (EDIT: and if so, what are they)?

• Are there any examples where the probability doesn't change after seeing the particular interval $[a,b]$ (i.e. the probability that the true parameter lies in $[a,b]$ is still 95%)?

• How (and why) does the probability change in general after seeing the computer spit out $[a,b]$?

[Edit]

Thanks for all the great answers and helpful discussions!

Answer 1

I think the fundamental problem is that frequentist statistics can only assign a probability to something that can have a long run frequency. Whether the true value of a parameter lies in a particular interval or not doesn't have a long run frequency, becuase we can only perform the experiment once, so you can't assign a frequentist probability to it. The problem arises from the definition of a probability. If you change the definition of a probability to a Bayesian one, then the problem instantly dissapears as you are no longer tied to discussion of long run frequencies.

See my (rather tounge in cheek) answer to a related question here:

"A Frequentist is someone that believes probabilies represent long run frequencies with which events ocurr; if needs be, he will invent a fictitious population from which your particular situation could be considered a random sample so that he can meaningfully talk about long run frequencies. If you ask him a question about a particular situation, he will not give a direct answer, but instead make a statement about this (possibly imaginary) population."

In the case of a confidence interval, the question we normally would like to ask (unless we have a problem in quality control for example) is "given this sample of data, return the smallest interval that contains the true value of the parameter with probability X". However a frequentist can't do this as the experiment is only performed once and so there are no long run frequencies that can be used to assign a probability. So instead the frequentist has to invent a population of experiments (that you didn't perform) from which the experiment you did perform can be considered a random sample. The frequentist then gives you an indirect answer about that fictitious population of experiments, rather than a direct answer to the question you really wanted to ask about a particular experiment.

Essentially it is a problem of language, the frequentist definition of a popuation simply doesn't allow discussion of the probability of the true value of a parameter lying in a particular interval. That doesn't mean frequentist statistics are bad, or not useful, but it is important to know the limitations.

Regarding the major update

I am not sure we can say that "Before we calculate a 95% confidence interval, there is a 95% probability that the interval we calculate will cover the true parameter." within a frequentist framework. There is an implicit inference here that the long run frequency with which the true value of the parameter lies in confidence intervals constructed by some particular method is also the probability that that the true value of the parameter will lie in the confidence interval for the particular sample of data we are going to use. This is a perfectly reasonable inference, but it is a Bayesian inference, not a frequentist one, as the probability that the true value of the parameter lies in the confidence interval that we construct for a particular sample of data has no long run freqency, as we only have one sample of data. This is exactly the danger of frequentist statistics, common sense reasoning about probability is generally Bayesian, in that it is about the degree of plausibility of a proposition.

We can however "make some sort of non-frequentist argument that we're 95% sure the true parameter will lie in [a,b]", that is exactly what a Bayesian credible interval is, and for many problems the Bayesian credible interval exactly coincides with the frequentist confidence interval.

"I don't want to make this a debate about the philosophy of probability", sadly this is unavoidable, the reason you can't assign a frequentist probability to whether the true value of the statistic lies in the confidence interval is a direct consequence of the frequentist philosophy of probability. Frequentists can only assign probabilities to things that can have long run frequencies, as that is how frequentists define probability in their philosophy. That doesn't make frequentist philosophy wrong, but it is important to understand the bounds imposed by the definition of a probability.

"Before I've entered the password and seen the interval (but after the computer has already calculated it), what's the probability that the interval will contain the true parameter? It's 95%, and this part is not up for debate:" This is incorrect, or at least in making such a statement, you have departed from the framework of frequentist statistics and have made a Bayesian inference involving a degree of plausibility in the truth of a statement, rather than a long run frequency. However, as I have said earlier, it is a perfectly reasonable and natural inference.

Nothing has changed before or after entering the password, because niether event can be assigned a frequentist probability. Frequentist statistics can be rather counter-intuitive as we often want to ask questions about degrees of plausibility of statements regarding particular events, but this lies outside the remit of frequentist statistics, and this is the origin of most misinterpretations of frequentist procedures.

Answer 2

Major update, major new answer. Let me try to clearly address this point, because it's where the problem lies:

"If you argue that "after seeing the interval, the notion of probability no longer makes sense", then fine, let's work in an interpretation of probability in which it does make sense."

The rules of probability don't change but your model for the universe does. Are you willing to quantify your prior beliefs about a parameter using a probability distribution? Is updating that probability distribution after seeing the data a reasonable thing to do? If you think so then you can make statements like $P(\theta\in [L(X), U(X)]| X=x)$. My prior distribution can represent my uncertainty about the true state of nature, not just randomness as it is commonly understood - that is, if I assign a prior distribution to the number of red balls in an urn that doesn't mean I think the number of red balls is random. It's fixed, but I'm uncertain about it.

Several people including I have said this, but if you aren't willing to call $\theta$ a random variable then the statement $P(\theta\in [L(X), U(X)]| X=x)$ isn't meaningful. If I'm a frequentist, I'm treating $\theta$ as a fixed quantity AND I can't ascribe a probability distribution to it. Why? Because it's fixed, and my interpretation of probability is in terms of long-run frequencies. The number of red balls in the urn doesn't ever change. $\theta$ is what $\theta$ is. If I pull out a few balls then I have a random sample. I can ask what would happen if I took a bunch of random samples - that is to say, I can talk about $P(\theta\in [L(X), U(X)])$ because the interval depends on the sample, which is (wait for it!) random.

But you don't want that. You want $P(\theta\in [L(X), U(X)]| X=x)$ - what's the probability that this interval I constructed with my observed (and now fixed) sample contains the parameter. However, once you've conditioned on $X=x$ then to me, a frequentist, there is nothing random left and the statement $P(\theta\in [L(X), U(X)]| X=x)$ doesn't make sense in any meaningful way.

The only principled way (IMO) to make a statement about $P(\theta\in [L(X), U(X)]| X=x)$ is to quantify our uncertainty about a parameter with a (prior) probability distribution and update that distribution with new information via Bayes Theorem. Every other approach I have seen is a lackluster approximation to Bayes. You certainly can't do it from a frequentist perspective.

That isn't to say that you can't evaluate traditional frequentist procedures from a Bayesian perspective (often confidence intervals are just credible intervals under uniform priors, for example) or that evaluating Bayesian estimators/credible intervals from a frequentist perspective isn't valuable (I think it can be). It isn't to say that classical/frequentist statistics is useless, because it isn't. It is what it is, and we shouldn't try to make it more.

Do you think it's reasonable to give a parameter a prior distribution to represent your beliefs about the universe? It sounds like it from your comments that you do; in my experience most people would agree (that's the little half-joke I made in my comment to @G. Jay Kerns's answer). If so, the Bayesian paradigm provides a logical, coherent way to make statements about $P(\theta\in [L(X), U(X)]| X=x)$. The frequentist approach simply doesn't.

Answer 3

OK, now you're talking! I've voted to delete my previous answer because it doesn't make sense with this major-updated question.

In this new, updated question, with a computer that calculates 95% confidence intervals, under the orthodox frequentist interpretation, here are the answers to your questions:

1. No.
2. No.
3. Once the interval is observed, it is not random any more, and does not change. (Maybe the interval was $[1,3]$.) But $\theta$ doesn't change, either, and has never changed. (Maybe it is $\theta = 7$.) The probability changes from 95% to 0% because 95% of the intervals the computer calculates cover 7, but 100% of the intervals $[1,3]$ do NOT cover 7.

(By the way, in the real world, the experimenter never knows that $\theta = 7$, which means the experimenter can never know whether the true probability $[1,3]$ covers $\theta$ is zero or one. (S)he only can say that it must be one or the other.) That, plus the experimenter can say that 95% of the computer's intervals cover $\theta$, but we knew that already.

The spirit of your question keeps hinting back to the observer's knowledge, and how that relates to where $\theta$ lies. That (presumably) is why you were talking about the password, about the computer calculating the interval without your seeing it yet, etc. I've seen in your comments to answers that it seems unsatisfactory/unseemly to be obliged to commit to 0 or 1, after all, why couldn't we believe it is 87%, or $15/16$, or even 99%??? But that is exactly the power - and simultaneously the Achilles' heel - of the frequentist framework: the subjective knowledge/belief of the observer is irrelevant. All that matters is a long-run relative frequency. Nothing more, nothing less.

As a final BTW: if you change your interpretation of probability (which you intentially have elected not to do for this question), then the new answers are:

1. Yes.
2. Yes.
3. The probability changes because probability = subjective knowledge, or degree of belief, and the knowledge of the observer changed. We represent knowledge with prior/posterior distributions, and as new information becomes available, the former morphs into the latter (via Bayes' Rule).

(But for full disclosure, the setup you describe doesn't match the subjective interpretation very well. For instance, we usually have a 95% prior credible interval before even turning on the computer, then we fire it up and employ the computer to give us a 95% posterior credible interval which is usually considerably skinnier than the prior one.)

Answer 4

The reason that the confidence interval doesn't specify "the probability that the true parameter lies in the interval" is because once the interval is specified, the paramater either lies in it or it doesn't. However, for a 95% confidence interval for example, you have a 95% chance of creating a confidence interval that does contain the value. This is a pretty difficult concept to grasp, so I may not be articulating it well. See http://frank.itlab.us/datamodel/node39.html for further clarification.

Answer 5

The way you pose the problem is a little muddled. Take this statement: Let $E$ be the event that the true parameter falls in the interval $[a,b]$. This statement is meaningless from a frequentist perspective; the parameter is the parameter and it doesn't fall anywhere, it just is. P(E) is meaningless, P(E|C) is meaningless and this is why your example falls apart. The problem isn't conditioning on a set of measure zero either; the problem is that you're trying to make probability statements about something that isn't a random variable.

A frequentist would say something like: Let $\tilde E$ be the event that the interval $(L(X), U(X))$ contains the true parameter. This is something a frequentist can assign a probability to.

Edit: @G. Jay Kerns makes the argument better than me, and types faster, so probably just move along :)

Answer 6

I don't think a frequentist can say there is any probability of the true (population) value of a statistic lying in the confidence interval for a particular sample. It either is, or it isn't, but there is no long run frequency for a particular event, just the population of events that you would get by repeated performance of a statistical procedure. This is why we have to stick with statements such as "95% of confidence intervals so constructed will contain the true value of the statistic", but not "there is a p% probability that the true value lies in the confidence interval computed for this particular sample". This is true for any value of p, it simply isn't possible withing the frequentist definition of what a probability actually is. A Bayesian can make such a statement using a credible interval though.

Answer 7

In frequentist statistics, the event $E$ is fixed -- the parameter either lies in $[a, b]$ or it doesn't. Thus, $E$ is independent of $C$ and $C'$ and so both $P(E|C) = P(E)$ and $P(E|C') = P(E)$.

(In your argument, you seem to think that $P(E|C) = 1$ and $P(E|C') = 0$, which is incorrect.)

Answer 8

I'll throw in my two cents (maybe redigesting some of the former answers). To a frequentist, the confidence interval itself is in essence a two-dimensional random variable: if you would redo the experiment a gazillion times, the confidence interval you would estimate (i.e.: calculate from your newly found data each time) would differ each time. As such, the two boundaries of the interval are random variables.

A 95% CI, then, means nothing more than the assurance (given all your assumptions leading to this CI are correct) that this set of random variables will contain the true value (a very frequentist expression) in 95% of the cases.

You can easily calculate the confidence interval for the mean of 100 draws from a standard normal distribution. Then, if you draw 10000 times 100 values from that standard normal distribution, and each time calculate the confidence interval for the mean, you will indeed see that 0 is in there about 9500 times.

The fact that you have created a confidence interval just once (from your actual data) does indeed reduce the probability of the true value being in that interval to either 0 or 1, but it doesn't change the probability of the confidence interval as a random variable to contain the true value.

So, bottom line: the probability of any (i.e. on average) 95% confidence interval containing the true value (95%) doesn't change, and neither does the probability of a particular interval (CI or whatever) for containing the true value (0 or 1). The probability of the interval the computer knows but you don't is actually 0 or 1 (because it is a particular interval), but since you don't know it (and, in a frequentist fashion, are unable to recalculate this same interval infinitely many times again from the same data), all you have to go for is the probability of any interval.

Answer 9

If I say the probability the Knicks scored between xbar - 2sd(x) and xbar + 2sd(x) is about .95 in some given game in the past, that is a reasonable statement given some particular distributional assumption about the distribution of basketball scores. If I gather data about the scores given some sample of games and calculate that interval, the probability that they scored in that interval on some given day in the past is clearly zero or one, and you can google the game result to find out. The only notion of it maintaining non-zero or one probability to the frequentist comes from repeated sampling, and the realization of interval estimation of a particular sample is the magic point where either it happened or it didn't given the interval estimate of that sample. It isn't the point where you type in the password, it is the point where you decided to take a single sample that you lose the continuity of possible probabilities.

This is what Dikran argues above, and I have voted up his answer. The point when repeated samples are out of the consideration is the point in the frequentist paradigm where the non-discrete probability becomes unobtainable, not when you type in the password as in your example above, or when you google the result in my example of the Knicks game, but the point when your number of samples =1.

Answer 10

There are so many long explanations here that I don't have time to read them. But I think the answer to the basic question can be short and sweet. It is the difference between a probability that is unconditional on the data. The probability of 1-alpha before collecting the dats is the probability that the well-defined procedure will include the parameter. After you have collected the data and know the specific interval that you have generated the interval is fixed and so since the parameter is a constant this conditional probability is either 0 or 1. But since we don't know the actual value of the parameter even after collecting the data we don't know which value it is.

Extension of the post by Michael Chernick copied form comments:

there is a pathological exception to this which can be called perfect estimation. Suppose we have a first order autoregressive process given by X(n)=pX(n-1) + en. It is stationary so we know p is not 1 or -1 and is < 1 in absolute value. Now the en are independent identically distributed with a mixed distribution there is a positive probability q that en= 0

There is a pathological exception to this which can be called perfect estimation. Suppose we have a first order autoregressive process given by X(n)=pX(n-1) + en. It is stationary so we know p is not 1 or -1 and is < 1 in absolute value.

Now the en are independent identically distributed with a mixed distribution there is a positive probability q that en=0 and with probability 1-q it has an absolutely continuous distribution (say that the density is non zero in an interval bounded away from 0. Then collect data from the time series sequentially and for each successive pair of values estimate p by X(i)/X(i-1). Now when ei = 0 the ratio will equal p exactly.

Because q is greater than 0 eventually the ratio will repeat a value and that value has to be the exact value of the parameter p because if it is not the value of ei which is not 0 will repeat with probability 0 and ei/x(i-1) will not repeat.

So the sequential stopping rule is to sample until the ratio repeats exactly then use the repeated value as the estimate of p. Since it is p exactly any interval you construct that is centered at this estimate has probability 1 of including the true parameter. Although this is a pathological example that is not practical there do exist stationary stochastic processes with the properties that we require for the error distribution