13
$\begingroup$

Note: apologies in advance if this is a duplicate, I didn't find a similar q in my search

Say we have a true parameter p. A confidence interval C(X) is a RV that contains p, say 95% of the time. Now suppose we observe X and compute C(X). The common answer seems to be that it is incorrect to interpret this as having a "95% chance of containing p" since it "either does or it doesn't contain p"

However, let's say I pick a card from the top of a shuffled deck and leave it face down. Intuitively I think of the probability of this card of being the Ace of Spades as 1/52, even though in reality "it either is or it isn't the Ace of Spades." Why can't I apply this reasoning to the example of the confidence interval?

Or if it is not meaningful to talk of the "probability" of the card being the ace of spades since it "is or it isn't", I would still lay 51:1 odds that it isn't the ace of spades. Is there another word to describe this information? How is this concept different than "probability"?

edit: Maybe to be more clear, from a bayesian interpretation of probability, if I'm told that a random variable contains p 95% of the time, given the realization of that random variable (and no other information to condition on) is it correct to say the random variable has a 95% probability of containing p?

edit: also, from a frequentist interpretation of probability, let's say the frequentist agrees not to say anything like "there is a 95% probability that the confidence interval contains p". Is it still logical for a frequentist to have a "confidence" that the confidence interval contains p?

Let alpha be the significance level and let t = 100-alpha. K(t) be the frequentist's "confidence" that the confidence interval contains p. It makes sense that K(t) should be increasing in t. When t = 100%, the frequentist should have certainty (by definition) that the confidence interval contains p, so we can normalize K(1) = 1. Similarly, K(0) = 0. Presumably K(0.95) is somewhere between 0 and 1 and K(0.999999) is greater. In what way would the frequentist consider K different from P (the probability distribution)?

$\endgroup$
  • 1
    $\begingroup$ Indeed, consider a coin flip where the coin rolls under a table, out of view and we consider the event that the coin landed on heads. At first glance this seems to be very similar to the CI issue - clearly either the event happened or it didn't. Yet in the coin flip case many (perhaps even most) frequentists seem perfectly happy to assign a notional probability, (say $p$) to the unobserved coin having ended up on heads, while backing away from saying the same thing about the random interval containing the parameter. To me there seems to be an inconsistency. $\endgroup$ – Glen_b Jun 25 '15 at 2:29
  • $\begingroup$ @Glen_b Frequentists in the unobserved, dropped coin scenario apply counterfactual reasoning to say, not that the actual face value of the coin is "random" (though it's unobserved), but that we may generalize any observed outcome to other potential outcomes in this dropped coin and calculate probabilities. As far as the probability for the actual face value of the coin, it either is or isn't heads, there is no probability. The $p$ is saved for the counterfactual construction of this setting. $\endgroup$ – AdamO Jun 25 '15 at 18:51
  • $\begingroup$ @Glen_b: I agree, see my question here: stats.stackexchange.com/questions/233588/… $\endgroup$ – vonjd Sep 6 '16 at 17:53
  • $\begingroup$ @vonjd to what extent is your question there not simply a duplicate of the first paragraph after the opening "Note:" here? $\endgroup$ – Glen_b Sep 6 '16 at 23:09
  • $\begingroup$ @Glen_b: To be honest I wasn't aware of this question when I posted mine and they certainly overlap. Yet I think they are not duplicates because mine is more generally concerned with using probabilities for hidden outcomes (which would have consequences for confidence intervals) whereas this one is purely aiming at confidence intervals. But if you think that mine is a duplicate feel free to close it. $\endgroup$ – vonjd Sep 7 '16 at 5:20
7
$\begingroup$

I think lots of conventional accounts of this matter are not clear.

Lets say you take a sample of size $100$ and get a $95\%$ confidence interval for $p$.

Then you take another sample of $100$, independent of the first, and get another $95\%$ confidence interval for $p$.

What changes is the confidence interval; what does not change is $p$. That means that in frequentist methods, one says the confidence interval is "random" but $p$ is "fixed" or "constant", i.e. not random. In frequentist methods, such as the method of confidence intervals, one assigns probabilities only to things that are random.

So $\Pr(L<p<U)=0.95$ and $(L,U)$ is a confidence interval. ($L=$ "lower" and $U=$ "upper".) Take a new sample and $L$ and $U$ change but $p$ does not.

Let's say in a particular instance you have $L=40.53$ and $U=43.61$. In frequentist methods one would not assign a probability to the statement $40.53<p<43.61$, other than a probability of $0$ or $1$, becuase nothing here is random: $40.53$ is not random, $p$ is not random (since it won't change if we take a new sample), and $43.61$ is not random.

In practice, people do behave as if they're $95\%$ sure that $p$ is between $40.53$ and $43.61$. And as a practical matter, that may often make sense. But sometimes it doesn't. One such case is if numbers as large as $40$ or more are known in advance to be improbable, or if they are known to be highly probable. If one can assign some prior probability distribution to $p$, one uses Bayes theorem to get a credible interval, which may differ from the confidence interval because of prior knowledge of which ranges of values of $p$ are probable or improbable. It can also actually happen that the data themselves --- the things that change if a new sample is taken, can tell you that $p$ is unlikely to be, or even certain not to be, as big as $40$. That can happen even in cases in which the pair $(L,U)$ is a sufficient statistic for $p$. That phenomenon can be dealt with in some instances by Fisher's method of conditioning on an ancillary statistic. An example of this last phenomenon is when the sample consists of just two independent observations that are uniformly distributed in the interval $\theta\pm1/2$. Then the interval from the smaller of the two observations to the larger is a $50\%$ confidence interval. But if the distance between them is $0.001$, it would be absurd to be anywhere near $50\%$ sure that $\theta$ is between them, and if the distance is $0.999$, one would reasonably be almost $100\%$ sure $\theta$ is between them. The distance between them would be the ancillary statistic on which one would condition.

$\endgroup$
  • $\begingroup$ Thanks Michael that makes a lot of sense. Let's suppose in your example that we have a particular (L,U) but the values are not known to us. All we know is that it's the realization of a the 95% confidence interval random variable. Without any prior on the parameter or any other information, would it be fair to lay 19:1 odds that (L,U) contains the parameter? If a frequentist is willing to do this, but not call his "willingness to lay 19:1 odds that it contains the parameter" a "probability", what would we call it? $\endgroup$ – applicative_x Jun 25 '15 at 4:28
  • $\begingroup$ Yes, that probability is $0.95$. Certainly within frequentist methods one can say that in a state of ignorance of $(L,U)$ the probability is $0.95$ that that interval contains $p$. But when one has particular values, which are not random, the frequentist will not assign a probabliity other than $0$ or $1$ to the statement, since the known values of $L$ and $U$ are not random. $\endgroup$ – Michael Hardy Jun 25 '15 at 4:34
3
$\begingroup$

The textbook definition of a $100 \times (1-\alpha)$% confidence interval is:

An interval which, under many independent replications of the study under ideal conditions, captures the replicated effect measurement $100 \times (1-\alpha)$% of the time.

Probability, to frequentists, comes from the notion of "rewinding time and space" to replicate findings, as if an infinite number of copies of the world were created to assess a scientific finding again and again and again. So a probability is a frequency exactly. For scientists, this is a very convenient way to discuss findings, since the first principle of science is that studies must be replicable.

In your card example, the confusion for Bayesians and Frequentists is that the frequentist does not assign a probability to the face value of the particular card you have flipped from the deck whereas a Bayesian would. The frequentist would assign the probability to a card, flipped from the top of randomly shuffled deck. A Bayesian is not concerned with replicating the study, once the card is flipped, you now have 100% belief about what the card is and 0% belief that it could take any other value. For Bayesians, probability is a measure of belief.

Note that Bayesians do not have confidence intervals for this reason, they summarize uncertainty with credibility intervals.

$\endgroup$
  • $\begingroup$ Thanks for the response. In the card example, wouldn't both the bayesian and frequentist agree that 51:1 is fair odds that the card is the ace of spades? Similarly, for the realization of a 95% confidence interval (and no other information), wouldn't both lay 19:1 odds that it contains the true parameter? In that sense, could a bayesian interpret the 95% confidence interval as having a 95% chance of containing the true parameter? $\endgroup$ – applicative_x Jun 25 '15 at 1:09
  • $\begingroup$ @applicative_x What about a pinochle deck? You are considering the use of prior information. The frequentist may only hypothesize that the probability is $p=1/52$ and only use the card's face valueto inform whether this experiment was consistent or inconsistent with that hypothesis. The validity of any type of interval estimate (credibility or confidence) depends on unverifiable assumptions. There is no such thing as a true parameter, this is a dangerous way of thinking about science. Bayesians do not play with confidence intervals per the earlier definition. Reread the answer. $\endgroup$ – AdamO Jun 25 '15 at 1:13
  • $\begingroup$ Thanks Adam, I think I am still confused. Let's suppose I know (by looking at the cards) that a 52 card deck is standard. I shuffle the deck and pick out the top 10 cards without looking at them. Couldn't I define the "true parameter" in this case to be the number of red cards? Then regardless of bayesian vs frequentist there is a "true parameter." If I am allowed to pick 7 cards at random I could also imagine constructing a confidence interval for the #of red cards out of my 10. $\endgroup$ – applicative_x Jun 25 '15 at 1:29
  • 1
    $\begingroup$ A Bayesian doesn't have to believe there is no such thing as a true value of a parameter. Bayesianism just means assigning probabilities to statements that are uncertain, regardless of whether they are random. A Bayesian can assign probability $1/2$ to the statement that there was life on Mars a billion years ago. A frequentist cannot do that, since one cannot say that that happened in half of all cases. Nothing in that says a Bayesian cannot believe there is a true answer to the question whether there was such life on Mars. See also my posted answer to your question. $\endgroup$ – Michael Hardy Jun 25 '15 at 4:18
  • 1
    $\begingroup$ @AdamO : I find your comments cryptic. 'of what utility is the notion of "truth"' is a change of subject. "We think of truth as immutable." So "we" means you and who else, and what is the relevance of what they think? "No scientist would ever go about collecting data for the sake of verifying something that's already known." That seems like another change of subject. Then there follow some comments on frequentists and Bayesians. I don't feel like guessing what you're trying to say. $\endgroup$ – Michael Hardy Jun 26 '15 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.