12
$\begingroup$

I have a physician friend who asks me questions about stats. He gets confused about stuff, e.g., the definition of a confidence interval (CI) and its intricacies. For example, he finds the following confusing: Denote the parameter of interest $\theta$. Before data are drawn, $$ \mathbb{P}(\theta \in CI) = 1-\alpha. $$

However, once data are drawn there isn't any probability anymore and the true parameter is either inside the CI or not.

I told my friend that it is probably OK for him to think that "the parameter is inside the CI with probability $1-\alpha$". But I might be wrong. So I am looking for an example:

When is it important for a practitioner (physician, in this case) to fully understand the definition of a CI?

An example where misunderstanding CIs may lead to wrong clinical decisions would be ideal.

$\endgroup$
12
  • 5
  • 5
    $\begingroup$ Re "no probability any more:" that is not so. If it were correct, then one should be able either to state definitely whether the interval covers the parameter. $\endgroup$
    – whuber
    Apr 15 at 3:13
  • 6
    $\begingroup$ @whuber: I don't quite follow your logic. The CI either contains the parameter or doesn't, and in this sense there "is no probability" any more. But we still don't know whether it does. $\endgroup$ Apr 15 at 7:26
  • 3
    $\begingroup$ @Yair The "probability statement" is being made about the interval, not the parameter. $\endgroup$
    – whuber
    Apr 15 at 12:11
  • 5
    $\begingroup$ @Graham The Bayes/Frequentist/Whatever approach is separate from this. If we cannot make probability statements about completed events, then we have destroyed all possibility of modeling data probabilistically. Such confusion of temporal sequence, knowledge, and probability models is counterproductive. $\endgroup$
    – whuber
    Apr 15 at 12:13

11 Answers 11

13
$\begingroup$

I think the interpretation you provide is harmless in most basic research scenarios -- wrong, but ultimately harmless $^1$. Indeed, I have been known to provide this interpretation for non-technical stakeholders just to avoid having to walk them through the idea of repeated sampling.

However, if you do use this interpretation you need to be sure to combat misinterpretations of the CI as a prediction interval. For example, were you to talk about a confidence interval for the mean in this way, and since the mean is used as a prediction, the less statistically inclined may think that "There is 1-a probability the next observation is in my CI" which would be both wrong and harmful.

More to your question, so long as the physician is just doing basic research -- running regressions, making table 2, publishing the paper -- then this interpretation is fine. As soon as decisions and policy are being changed, then I would probably insist on some more nuanced understanding of the machinery being used to make those decisions since there is more at risk.


$^{1.}$ Actually not even that wrong, depending on what school of thought you subscribe to. I am told Confidence, Likelihood, Probability: Statistical Inference with Confidence Distributions explains this perspective more thoroughly, but I admit I have not read it myself.

$\endgroup$
5
  • $\begingroup$ The physician should collaborate with a medical statistician to help ensure the statistics are correct and correctly presented. Confidence intervals and credible intervals can often be safely confused because there is a prior for which they are numerically the same. However you need to know the difference for the cases where confusing them is not benign, especially for safety critical applications , like medicine. $\endgroup$ Apr 15 at 11:01
  • 2
    $\begingroup$ @DikranMarsupial I think you are holding a very high bar here for statistics and statistical practice in medicine -- which I anticipate is well meaning, but unfortunately unfeasible in most scenarios. There is a happy medium where good research can be done without having perfect understanding of exactly what the estimators are. $\endgroup$ Apr 15 at 17:40
  • 1
    $\begingroup$ I don't see it as that high a standard, as I said elsewhere CIs are a fairly basic concept that ought to be basic competence for practitioners, but I do agree that is not how things actually are. Kudos to the practitioner involved in this case for not being comfortable with their understanding and seeking advice. That is far better than being overly confident in, shall we say, "less than perfect" understanding. The real problem is that CIs are only a rather indirect answer to the question the practitioner really wants answered. $\endgroup$ Apr 15 at 17:51
  • 3
    $\begingroup$ @DikranMarsupial In my experience practitioners would be happy to have a probability statement for the true parameter, and they would be happy to get this without having to provide a prior. Also they don't want to know that the true parameter is defined within a model which is "wrong", as all models are, so the "true parameter" doesn't really exist. Sometimes we better tell practitioners that we can't provide what they want than pretending to do so. $\endgroup$ Apr 15 at 22:30
  • $\begingroup$ @ChristianHennig Completely agree with the last sentence! My own view is that we should try to use the tools that answer our questions most directly - if we need a prior to do so, then we should do that (and be aware/discuss potential issues). It is not as if frequentist statistics is completely free of prior knowledge/subjectivity - it is how we ought to decide on appropriate significance levels etc. I think the burden should be more on the writer of the paper than the reader as that would be a more efficient approach. $\endgroup$ Apr 16 at 8:08
6
$\begingroup$

Greenland has proposed to rename confidence intervals to "compatibility intervals", which he defines as "the values of the parameter which, when combined with the background assumptions, produce a test model that is 'highly compatible' with the data". This is a nice way to report the values computed as the confidence interval, without using the often-confusing term "confidence".

$\endgroup$
7
  • 2
    $\begingroup$ E.T. Jaynes gave an example of a correctly constructed confidence interval that is certain not to contain the true value (I think it is the paper mentioned here: stats.stackexchange.com/questions/2356/… ), so I'm not sure "highly compatible with the data" is any better. I quite like "confidence interval" - the fact that frequentist terminology avoids mentioning probabilities is itself a statement. $\endgroup$ Apr 15 at 12:27
  • 2
    $\begingroup$ @DikranMarsupial Does the Jaynes example have a minimal dataset? From my reading I would say that it is relatively common for seeming counter-examples to consist of severely contrived conditions and less data than parameters. They do not in any way show that methods fall apart in more conventional circumstances. See here for an important example: arxiv.org/pdf/1507.08394.pdf $\endgroup$ Apr 15 at 20:38
  • 1
    $\begingroup$ @Harvey Your response begs the question of whether there would be circumstances where it would be important that a practitioner might interpret a compatibility interval as being a confidence interval? $\endgroup$ Apr 15 at 20:42
  • 1
    $\begingroup$ This paper by Greenland et al. is more pertinent to the point made here, I think. The one currently linked to is also fantastic. $\endgroup$ Apr 15 at 21:06
  • 2
    $\begingroup$ @MichaelLew It is indeed a contrived example, but it is intended to make a point clearly. The fact that you can contrive such an example shows that the misinterpretation of confidence intervals is a misinterpretation. I found it a very useful example in clarifying the distinction between a confidence interval and a credible interval, which makes me appreciate confidence intervals more. Nobody is saying that confidence intervals "fall apart", that is only if you insist on interpreting them as a direct answer to a question they are not intended to answer directly. $\endgroup$ Apr 16 at 8:02
6
$\begingroup$

Let's say we test a new drug compared with a placebo in a randomized controlled trial to see whether the drug lowers blood pressure. After 12 weeks on the drug, the systolic blood pressure is on average 30 mmHg lower (95% CI 15 to 45 mmHg). Let's look at 4 different scenarios:

  1. The drug we tested was a homeopathic preparation that was so diluted that not a single molecule of the original substance should remain (or we alternatively realize that we accidentally gave placebo to everyone).
  2. This was the first small trial for this drug in patients, the mode of action is completely novel and we know that new drugs in Phase 1b/2a have a success rate of about 10% (with many of the 90% that don't make it failing due to a lack of efficacy).
  3. This was a small Phase 4 study of a drug that was shown to lower blood pressure by 10 mmHg (95% CI 7.5 to 12.5 mmHg) based on a meta-analysis of two very large Phase 3 studies in the same population and a similar study design. While the Phase 3 studies included countries such as Belgium, Portugal, Norway, Finland and Germany, this new study involves the Netherlands, Spain, Sweden, Denkmark and Austria.
  4. This was the second Phase 3 study for a drug and the first Phase 3 study already had a point estimate of 31 mmHg blood pressure lowering (95% CI 14 to 47 mmHg). The drug is also from a class of drugs, for which 3 others drugs are already approved and showed similar effect sizes in their Phase 3 studies.

If you believe there's a 97.5% probability the true blood pressure lowering effect of the drug compared with placebo in the study population is >15 mmHg in all the cases (and similarly exactly a 2.5% probability that it's > 45 mmHg, but in a sense that's less interesting), you run the risk of making a lot of bad medical and business decisions.

Most reasonable people would argue that in the first scenario, you should probably have near zero probability that the true treatment effect of the intervention is >15 mmHg, in the second & third scenarios it's higher but substantially below 97.5% (perhaps lower in the third scenario vs. the second, but one could disagree), and substantially above 97.5% in the fourth scenario.

You may argue that I'm going all Bayesian on you, but it's extremely rare that we don't have some prior knowledge. Arguably, a Bayesian perspective might allow you to claim the kind of "95% probability the true value is in the CI" statement, but that's only if an (almost) completely flat prior is correct (an effect of blood pressure lowering of near 299,792,458 is just as likely as one near 3.141592654 or $- 17.213 \times 10^{-231321}$). When we don't have absolutely no prior information, this is incorrect (and shrugging your shoulders and refusing to pin down the available prior information because it's too tedious to work out, is not an excuse, either).

$\endgroup$
5
  • $\begingroup$ your answer is correct, but does not answer the OP's question. In addition, you confuse alpha errors (false positives, when teh Null is true) with PPV (aka FDR). All a CI controls is alpha errors. In your examples, detecting a "significant effect when none was there. And the CI does this "perfectly"; you will have such a false positive (which implies null was true, i.e. effect was minimal) only 5% of the time. But that does not control PPV: if I have a significant result, is it a true positive (there was an effect) or a false positive. ...cont... $\endgroup$
    – jginestet
    Apr 15 at 19:28
  • $\begingroup$ To control PPV, you need to know the prevalence (or the prior probability $P(H_0)=1-P(H_a)$. And yes, it is Baeysian, but Bayes theorem applies even in frequentist statistics. You will get a false positive only $100*(1-\alpha)%$ of the time, but depending on $P(H_a)$ (as in your examples, where it goes from almost 0 to almost certainty), you should (or not) deduce that the drug "works". This is not an issue of wrong interpretation of the CI (just as it would not be a wrong interpretation of the p-value). It is another case of base-rate paradox (aka false positive paradox). ...cont... $\endgroup$
    – jginestet
    Apr 15 at 19:35
  • $\begingroup$ ...cont... A "significant" result is just that. It is not a "truly significant" result (that is where the problem comes). It depends on the ratio of false positives to true positives. $\alpha$ controls the false positives, but not the true positives: $\alpha, \beta, and P(H_a)$ control that... $\endgroup$
    – jginestet
    Apr 15 at 19:37
  • 1
    $\begingroup$ @jginestet It answer's the OP's question in my opinion. Is he not asking whether it ever matters when you interpret the type I error rate as the PPV? When you claim that "the parameter is inside the CI with probability $1−\alpha$" after seeing the data, you're exactly claiming that it has PPV $1−\alpha$, which as you agree is not the case. $\endgroup$
    – Björn
    Apr 16 at 8:23
  • 2
    $\begingroup$ +1 that's the right answer IMHO. I think it's funny that we (as statisticians) widely agree that a p-value cannot be interpreted as the probability of H0 being correct, but often allow for the same sloppy usage of confidence intervals. I particularly like the figure in nature.com/articles/506150a, which could probably nicely be modified to illustrate your examples regarding confidence intervals. $\endgroup$
    – LuckyPal
    Apr 17 at 13:44
3
$\begingroup$

There are already good answers, so the following is more of a commentary or counterquestion: but if you find the quasi-Bayesian interpretation of confidence intervals so much more intuitive, why not simply use a Bayesian approach all the way?

If you want an interval that can be interpretated as "the probability that $\theta$ is within this range is $(1-\alpha)\%$", why not simply calculate the statistical object that allows you to make such statements unambiguously?

$\endgroup$
5
  • 4
    $\begingroup$ +1--good response. But bear in mind that we don't control what other researchers do; and there are millions of papers and books out there that use CIs; that there will continue to be many more CIs created even should Bayesian analyses suddenly become overwhelmingly popular; and there are many standards, regulations, and even court precedents that rely on CIs. $\endgroup$
    – whuber
    Apr 16 at 15:30
  • 2
    $\begingroup$ +1 use the tool that gives the most direct answer to the question you want to ask. It would be better if more use were made of credible intervals, and the onus ought to be on the writer to use statistical tools that are least likely to be misunderstood, however I agree with @whuber so we need better statistical education for practitioners that goes beyond applying recipies from the statistics cookbook. $\endgroup$ Apr 16 at 15:45
  • 1
    $\begingroup$ I understand that many people use CIs, albeit mostly out of convention rather than technical reasons. Rather than asking "under what circumstances is it ok to use a Bayesian interpretation for my frequentist confidence intervals", I thought a better question might be "what feature of my problem prevents me from using a Bayesian approach, such that I can interpret the interval estimates in the way that seems intuitive?" $\endgroup$
    – Durden
    Apr 16 at 15:49
  • $\begingroup$ Surely the truth that nobody is mentioning here is that the desire for the probability you mention is not always satisfied by a posterior probability. If I am told a credible interval has a 95% 'chance' of containing the parameter value, what does this chance mean? I know it is not the common meaning from games of chance. $\endgroup$ Apr 16 at 23:35
  • $\begingroup$ There a few posts on how to interpret posterior probabilities. $\endgroup$
    – Durden
    Apr 17 at 5:30
3
$\begingroup$

What is the precise definition of a confidence interval?

I made a video about this aimed at non-statisticians: https://youtu.be/jrUrjv_yM0M

A brief summary:

  1. A procedure for calculating a 95% confidence intervals will result in [an interval containing the true parameter] 95% of the time.
  2. In theory, you could construct confidence intervals in multiple ways to satisfy (1), but all result in (slightly) different intervals.
  3. It is possible for one confidence interval to be shorter and completely contained in another interval, even though both were made with a valid 95% confidence procedure.
  4. Hence, the statement of probability is about the procedure used to create intervals, not about any particular interval itself.

Below is the correct interpretation visualized. Imagine performing 100 experiments and constructing 95% confidence intervals each time:

CI95

(On average) 95% of these intervals will contain the true value.


When does the precise definition of a confidence interval matter?

In my opinion, this is especially the case...

  • When it can be logically expected to differ substantially from a credible interval (strong prior beliefs, small data).
  • When it is used in subsequent calculations. In this case, it is important to know that the bounds of a given interval do not necessarily provide protection up to a specified probability.
  • When writing a scientific article. Misinterpretations propagate when authors are imprecise with language.
$\endgroup$
1
2
$\begingroup$

It's all about the expression. To put it very simple (because there is no more to it)

(shortcut: familiar with p-values? It's just the opposite)

  • A confidence interval is a "guess"

  • The guess is correct with probability 1 - $\alpha$

The crucial point is that the probability is associated with the "guess" that you make: whatever procedure you choose to make your guess, if repeated many times, the guess coming from the procedure will be correct a certain fraction of times.$^1$

That means, if you actually make a guess at something, you know, from trying it out often before, your guess will be right with a certain probability.

why does it matter?

Inference using data is about finding out, how was this data produced; it's about infering an underlying quantity of the data production mechanism (such as a natural constant, or the fact that medicin works or not). Your data is one out of infinitely many possible samples. But the value that we try to infer is already determined.

$^1$ That's the probability that "the procedure yields a guess that is correct", in short, "that your guess is correct"; in correct: the coverage of your interval.

$\endgroup$
4
  • 4
    $\begingroup$ I like this approach--but it doesn't seem to get us anywhere: isn't "your guess is correct" just an equivalent description of "team A won"? $\endgroup$
    – whuber
    Apr 16 at 13:50
  • $\begingroup$ Thanks for the feedback @whuber. yes, if the guess is correct. But that's a different situation, in this case, the interval covers it 100% of the time. Maybe think of it like a "bet", sure, if you bet is correct, that means that whatever you've bet on was true. But that's "once you won your bet". The important part is that the probability is associated with the guess to be true or not, and not with the outcome of the game. $\endgroup$
    – Mayou36
    Apr 17 at 15:04
  • $\begingroup$ This is often IMHO more clearly stated in terms of a confidence limit procedure, which is designed to yield a guess that covers the parameter at least $1-\alpha$ of the time, no matter what the parameter's value might be. I believe much of the confusion lurking in the original question comes down to a lack of distinction between the procedure and its outcome (just as if one failed to distinguish between the process of flipping a coin and the result of that flip). That distinction is apparent in your answer but "your guess is correct" doesn't really capture it. $\endgroup$
    – whuber
    Apr 17 at 15:07
  • 1
    $\begingroup$ I agree, it doesn't capture it too well. I've reformulated and removed the example until I come up with a better one :) $\endgroup$
    – Mayou36
    Apr 17 at 16:17
1
$\begingroup$

For example, he finds the following confusing: Denote the parameter of interest θ . Before data are drawn, P(θ∈CI)=1−α.

That is indeed confusing, to the point of communicating something false. Wording it that way makes it sound like we have a fixed CI, and $\theta$ is a random variable that can be in the CI or not. But in fact $\theta$ is fixed, and the CI is a random variable that can include $\theta$ or not. While "The CI includes $\theta$" and "$\theta$ is in the CI" are logically equivalent, they communicate different things, especially when we start talking about the probability of the events. When we say that we have a 95% CI, what we mean is that we have an interval that was generated by an algorithm that (given certain assumptions about the distribution) has a 95% probability of producing an interval that contains the true parameter.

Suppose you know someone who, when they tosses a horseshoe, gets it on the stake 95% of the time. If you can see the horseshoe after they throw it, but not the stake, then the horseshoe is a 95% CI for where the stake is.

However, once data are drawn there isn't any probability anymore and the true parameter is either inside the CI or not.

No. Not "The true parameter is either inside the CI or not", but rather "The CI either contains the true parameter or not."

An example where misunderstanding CIs may lead to wrong clinical decisions would be ideal.

Two issues (other than having a flawed model) are confusing P(A|B) and P(B|A), and failing to incorporate additional data.

What sort of situation would a physician rely on CIs? One I can think of is if there's a diagnostic test that estimates some quantity, and there's some cutoff for diagnosing a disease, and the test yields a CI that excludes that cutoff. One might then conclude that there's only a 5% change that the "real" value is past the cutoff. But remember that a 95% CI means that 95% of all of the CIs include the true value. CIs that exclude the cutoff are a subset of all CIs. So it doesn't necessarily follow that 95% of those CIs exclude the true value.

Going back to the example of someone throwing horseshoes that I gave earlier, if I tell you that they get the horseshoe on the stake 95% of the time, then if you have nothing other than this to go on, you can be confident that there is a good chance that it's on the stake now. But what if I then tell you that actually, they do 95% of their horseshoe throwing during the day, and get 100% of those throws onto the stake, but get 0% of their nighttime throws onto the stake? Now it's critical to know what time of day it is.

Similarly, if a patient has a bunch of symptoms that are pointing towards a disease, but one test gives a 95% CI that excludes the cutoff, those other symptoms shouldn't be ignored in favor of the one test. And if all you have is the test, you should think about whether you could gather further information, rather than just stopping at "95% chance of healthy!"

$\endgroup$
2
  • $\begingroup$ W.r.t. "Not 'The true parameter is either inside the CI or not', but rather 'The CI either contains the true parameter or not.'" & "While 'The CI includes $\theta$' and "$\theta$ is in the CI" are logically equivalent, they communicate different things": if there are supposed to be subtle linguistic cues here to tell us what's a random variable & what isn't, they're too subtle for me & undoubtedly for any reader who needs an explanation of confidence intervals. $\endgroup$ Apr 17 at 18:10
  • $\begingroup$ @Scortchi-ReinstateMonica Something being the subject of the verb conveys a more active role. The CI can contain the parameter or not. That something that CI is doing or not. There's a dependency on the CI, because it's the CI that's a random variable. $\endgroup$ Apr 22 at 2:12
0
$\begingroup$

I am going to give a non-medical take.

As an engineer, I can look at historic hourly temperatures at a site, and determine what a change in hardware would have given them if it had been in place in that period.

"If you had the economizer in last year you would have saved $\$$xyz on your utilities at a price of $\$$pqr"

Last year in weather is not a perfect indicator for this year in weather, so saying "you should expect to save $\$$xyz this year if you implement it" is a nearly always wrong. When you estimate the mean, then half the time the guess is too low, and the other half it is too high.

So a better approach is to bootstrap resample the temperatures for a few years, and look at the quantiles of the savings, and give a range to go with the central tendency. I like to say something to the effect of "your range should be between $\$$abc and $\$$def, and though the best single value to $\$$xyz, about half the time it is going to be too low and the other half of the time too high, but this is only true as long as the last few years are consistent indicators of the next few years."

That shows how it can be useful in practice, and turn something that is wrongly understood by the customer 100% of the time into something where you are not wrong about 95% of the time, given your caveat of historic consistency.

$\endgroup$
-1
$\begingroup$

Your question touches on the classic (should I say "purist"?) frequentist interpretation of CI's. @Demetri Panaos says that your interpretation is wrong (but maybe not so much, as his footnote states), but harmless. I will go one step further. Your interpretation is correct, and exactly equivalent to the "accepted" interpretation. So let me share how I explain this to my (undergrad) students (who seem to get it).

Let's start with the "official" interpretation. If I were to take a very large number of samples, and for each compute its associated CI (of whatever parameter $\theta$ I am interested in), I would get an equally very large number of CI's, all somehow different (because of the randomness of my variable). These CI's are such that $100\cdot(1-\alpha)%$ of them would actually contain the true population parameter $\theta$.

Now, let's continue from here. The fact is you do not have a very large number of CI's, but only the 1 from your experiment. But that 1 has been taken randomly out the very large number of (theoritical) CI's one could have obtained. So that one, single CI has a $100\cdot(1-\alpha)%$ chance of containing the true population parameter $\theta$ (because you randomly took it from a population which had a $100\cdot(1-\alpha)%$ probability of containing $\theta$).

Now your friend asks; "But after I got my sample, my single CI either contains $\theta$ or it does not". This is true. But we do not know $\theta$; there is uncertainty about its true value. Hence we can use probabilities to describe this uncertainty: there is a $100\cdot(1-\alpha)%$ probability that this CI contains the true $\theta$ (the CI is a random variable, from a population with $100\cdot(1-\alpha)%$ proportion of success).

When the purists get very upset is if you start saying "$\theta$ has a $100\cdot(1-\alpha)%$ of lying in the CI" (because $\theta$ is not a random variable, and hence talking about probabilities associated with $\theta$ is somehow incorrect; but the above sentence does not say that $\theta$ is random, only that its belonging to this 1 CI is random).

So the "correct" interpretation is "there is a $100\cdot(1-\alpha)%$ probability that this CI contains the true $\theta$", and an incorrect one is "$\theta$ has a $100\cdot(1-\alpha)%$ of lying in the CI". To which I say "distinction without a difference". The first sentence says "there is a probability that B contains A", the second says "there is a probability that A is contained in B". These 2 sentences are semantically exactly equivalent. One is in the active voice, the 2nd in the passive voice. While English teachers have taught us not to use the passive voice, this is math/statistics here, and both statements are logically 100% equivalent.

So not only is your interpretation harmless, it is just as valid. $P(\theta ∈ CI)=1−\alpha \Longleftrightarrow P(CI ∋ \theta)=1-\alpha$
Both express the exact same uncertainty about the relative appertenance of $\theta$ to CI.

And therefore, as you ask at the end of your question, "an example where this type of misunderstanding of CIs may lead to wrong clinical decisions" does not exist.
(there could be several other misunderstandings which could lead to wrong clinical decisions; e.g. confusing the $\alpha$ error rate -which is controlled by the CI, for the PPV, aka false discovery rate. But that is for another answer...)

$\endgroup$
15
  • 4
    $\begingroup$ Unfortunately, your "correct" interpretation is not correct. Sure, pre-sampling, the random CI may cover the parameter with, say, 95% probability. However, post sample, there is no randomness left so the only probability attached to the sample CI is 0, or 1. From a frequentist perspective, your uncertainty about the true value of a parameter is not enough to justify any other probability value. Hence the term "confidence" is used. $\endgroup$ Apr 15 at 7:43
  • 5
    $\begingroup$ This answer is incorrect (see stats.stackexchange.com/questions/26450/… ) it is possible to construct a valid p% confidence interval for a sample of data that you can be absolutely certain does not contain the true value. I think the error immediately follows "Now, let's continue from here." your probability refers to the imaginary population of experiments that you didn't perform, not the specific experiment that you actually did. $\endgroup$ Apr 15 at 10:55
  • 2
    $\begingroup$ Confusing confidence intervals with Bayesian credible intervals is often benign because there is a choice of prior for which the two are numerically the same. However practitioners need to understand the distinction for the cases where it is not benign, because otherwise they can fall into a pitfall without knowing it was there. It is not just a matter of purism - they are answers to different questions. $\endgroup$ Apr 15 at 10:58
  • 2
    $\begingroup$ This is Fisher's fiducial argument, isn't it? Neyman thought it's wrong but it refuses to die... (also in the literature). $\endgroup$ Apr 15 at 22:36
  • 2
    $\begingroup$ @jginestet Do you know about fiducial inference en.wikipedia.org/wiki/Fiducial_inference ? How is your interpretation different, "common sense" or not? (It's not so "common" by the way, as many disagree with this.) $\endgroup$ Apr 16 at 0:04
-1
$\begingroup$

For example, he finds the following confusing: Denote the parameter of interest $\theta$. Before data are drawn, $$ \mathbb{P}(\theta \in CI) = 1-\alpha. $$

However, once data are drawn the is no probability anymore and the true parameter is either inside the CI or not.

This is a misintepretation of the confidence interval, one which confuses things further.

The x% confidence interval is precisely defined with respect to observed data; it is the range of 'models' (or parameterisations, if you prefer) for an appropriately stated hypothesis type, which would fail to reject the null hypothesis at the x% significance level "given" the observed data.

So it doesn't make sense to talk about a confidence interval before observing data. A CI is not an expression of posterior probability for the presence of a parameter over an interval: the name for this statistic is "credible interval", which is a very different concept.

The statistical guarantee a CI makes instead, is that, if you were to repeat an experiment an infinite number of times, such that you obtain new data from the same population from which you calculated a new x% CI each time, then x% of the time your computed CIs would correctly contain the true population parameter.

One way to think about this is as follows: As someone else mentioned here, effectively, in the same way that the p-value is effectively a measure of compatibility between a given model and a given set of data, then the CI effectively gives you the range of models that are sufficiently compatible with the observed data. When you repeat an experiment and get slightly different data, you'll get a slightly different CI, because the range of compatible models will be slightly different. Every now and then, with low probability, your data might be an outlier, and atypical of the actual population, and obtained simply by chance. In that case, your CI may show a range of models that wouldn't actually be very compatible with the population. But this CI will be obtained rarely; as rarely as it is rare to get an atypical sample from the population by chance.

$\endgroup$
-1
$\begingroup$

Given the difficulty that experienced statistical practitioners have when interpreting confidence intervals (see answers and comments here for evidence), it seems to me unlikely that there would be circumstances where such misinterpretation by a medical practitioner would be important. Therefore my answer to the titular question is "never".

That then carries with it some doubts about whether confidence intervals should be used (and taught) as the go-to type of interval statement.

$\endgroup$
2
  • $\begingroup$ Misinterpretation of confidence intervals by medical practitioners might be consequential, see McCormack et al. (2013). $\endgroup$
    – Durden
    Apr 17 at 17:10
  • $\begingroup$ @Durden: That seems to be more about researchers reporting just 'significant' or 'not significant' rather than looking at confidence intervals.. $\endgroup$ Apr 17 at 18:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.