What does a confidence interval (vs. a credible interval) actually express? [duplicate]

Question

Possible Duplicate:
What, precisely, is a confidence interval?

Yes, similar questions have been asked before, but many of the answers seem contradictory and don't address my issue. (Or my perception of the issue.)

As mentioned many places, what most people will likely find intuitive when presented with an interval and a probability, is that it expresses how probable it is that the true value lies in this range. If told that an exit poll has a confidence interval of 60-70 with probability 0.95, a layman may (reasonably) expect that when exit polls have this result, the interval does actually include the true proportion 95 % of the time. Expressed mathematically:

$P(X\in[60,70]) = 0.95$

The problem is, this seems to be the correct interpretation of credible intervals, and a common misinterpretation of confidence intervals. From http://en.wikipedia.org/wiki/Confidence_interval:

A confidence interval does not predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained.

So what does a confidence interval mean, then? Wikipedia says:

A confidence interval with a particular confidence level is intended to give the assurance that, if the statistical model is correct, then taken over all the data that might have been obtained, the procedure for constructing the interval would deliver a confidence interval that included the true value of the parameter the proportion of the time set by the confidence level.

I find the wording mighty confusing, but I understand this as meaning that given each X, there is at least a 0.95 probability of getting a Y whose interval spans X:

$P_X(Y : X \in I_y) \ge 0.95$

This seems to be consistent with the explanation of confidence and credible intervals given by Keith Winstein here: What's the difference between a confidence interval and a credible interval? (The probability, given a cookie jar, of picking a cookie with a chip count whose interval spans that same cookie jar is at least 70 %)

If this understanding is correct, then I fail to see why confidence intervals make any practical sense at all. Each interval depends on other intervals in ways that are difficult to grasp, and does not in fact have any strong connection to the actual result of a sampling.

Can someone explain why this concept is so widespread? (I realize that using Bayesian probability to obtain the credible interval may not be desirable, but that doesn't necessarily make CIs a good alternative.)

Answer 1

Both Confidence Intervals and Credible Intervals represent our knowledge about an unknown parameter given the data and other assumptions. When using lay interpretations the 2 intervals are pretty similar (though I may have just made frequentists and Bayesians have common ground in being offended by my statement). The tricky part comes when getting into exact definitions.

The Bayesians can talk about the probability of the parameter being in the interval, but they have to use the Bayesian definition of probability which is basically that probability represents our knowledge about an unknown parameter (look familiar?). Note that I am not a Bayesian, so they may want to give a better definition than mine. This does not work if you try to use a frequentist understanding of probability.

The frequentist definition of probability talks about the frequency that an outcome will occure if repeated a bunch of times at random. So once the randomness is over we cannot talk about probability any more, so we use the term confidence to represent the idea of amount of uncertainty after the event has occured (frequentist confidence is similar to Bayesian probability). Before I flip a fair coin I have a probability of 0.5 of getting heads, but after the coin has been flipped and landed or been caught it either shows a heads or a tails so the probability is either 0% or 100%, that is why frequentists don't like saying "probability" after the random piece is over (Bayesians don't have this problem since probability to them represents our knowledge about something, not the proportion of actual outcomes). Before collecting the sample from which you will compute your confidence interval you have a 95% chance of getting a sample that will generate a Confidence Interval that contains the true value. But once we have a Confidence Interval, the true value is either inside of that interval or it is not, and it does not change.

Imagine that you have an urn with 95 white balls and 5 black balls (or a higher total number with the same proportion). Now draw one ball out completely at random and hold it in your hand without looking at it (if you are worried about quantum uncertainty you can have a friend look at it but not tell you what color it is). Now you either have a white ball or a black ball in your hand, you just don't know which. A Bayesian can say that there is a 95% probability of having a white ball because their definition of probability represents the knowledge that you drew a ball at random where 95% were white. The frequentists could say that they are 95% confident that you have a white ball for the same reasoning, but neither would claim that if you open your hand 100 times and look at the ball (without drawing a new ball) that you will see a black ball about 5 times and a white about 95 times (which is what would happen if there was a 95% frequentist probability of having a white ball). Now imagine that the white balls represent samples that would lead to a correct CI and black balls represent samples that would not.

You can see this through simulation, either using a computer to simulate data from a known distribution or using a small finite population where you can compute the true mean. If you take a bunch of sample and compute Confidence Intervals and Credible Intervals for each sample, then compute the true mean (or other parameter) you will see that about 95% of the intervals contain the true value (if you have used reasonable assumptions). But if you concentrate on a single interval from a single sample, it either contains the true value or it does not, and no matter how long you stare at that given interval, the true value is not going to jump in or out.

Answer 2

I never understood confidence intervals until I read, in the wikipedia article, that frequentist confidence interval bounds are random variables. Yes, one performs some computation on observed data to construct confidence intervals, but since the data are (assumed) random variables, the confidence intervals are also random variables.

So for $l, u$ to be a symmetric 95 % confidence interval on population parameter, $\theta$, say, it should be the case that $l$ is a random variable that is larger than $\theta$ with probability 2.5 %, and similarly $u$ is a random variable that is smaller than $\theta$ with probability 2.5 %. For whatever reason, this formulation is easier for me to grasp: there is some unknown population parameter; I draw a sample from the population; as part of my draw, 2 'degrees of freedom' are spent on flipping biased coins for whether the $l$ and $u$ will actually bound the unknown parameter; I compute $l$ and $u$ from my sample.

Answer 3

Fruequentist:

By constructing an 1-alpha confidence interval, 1-alpha confidence intervals generated this way will include the "true" population parameter. Here the population parameter is fixed and the boundaries of the confidence interval are random.

This, however, says nothing about the probability of a certain confidence interval including the population parameter after the fact. After the fact, the confidence interval either includes the population parameter or it does not. There is nothing probabilistic anymore. So one cannot say that the true parameter lies within the confidence interval {-0.34, 0.2} with a probability of 95%, but rather that 95% of the confidence intervals generated by the same procedure (random sampling etc.) will include the fixed population parameter.

A good animation illustrating confidence intervals can be found on Yihui Xie's Website, who is author of the animation package in R:

Animation of confidence intervals: http://animation.yihui.name/mathstat:confidence_interval

Animation Package on CRAN: http://cran.r-project.org/web/packages/animation/index.html

Bayesian

Bayesians on the other hand incorporate uncertainty/information about the population parameter in their statistics (this is the so called "prior"). So a credible interval might be better thought of as a region of highest subjective believe. "Based on the prior information and the data I believe to 1-alpha % that the interval {-0.34, 0.2} includes the parameter". Most of the times this subjective believe is based on ohter data.

Practical Use

Both intervals say something about the accuracy of our estimate. If you want to let the data speak for themselves, you could use confidence intervals or bayesian credible intervals with a uniform prior. If, however, you have strong prior information that you want to not only include in your discussion section but also in your statistics, I would use credible intervals. So to me the problem is more one of what you want and less one of interpretation.