Morey et al (2015) argue that confidence intervals are misleading and there are multiple bias related to understanding of them. Among others, they describe the precision fallacy as following:

The Precision fallacy
The width of a confidence interval indicates the precision of our knowledge about the parameter. Narrow confidence intervals show precise knowledge, while wide confidence errors show imprecise knowledge.

There is no necessary connection between the precision of an estimate and the size of a confidence interval. One way to see this is to imagine two researchers — a senior researcher and a PhD student — are analyzing data of $50$ participants from an experiment. As an exercise for the PhD student's benefit, the senior researcher decides to randomly divide the participants into two sets of $25$ so that they can each separately analyze half the data set. In a subsequent meeting, the two share with one another their Student's $t$ confidence intervals for the mean. The PhD student's $95\%$ CI is $52 \pm 2$, and the senior researcher's $95\%$ CI is $53 \pm 4$.

The senior researcher notes that their results are broadly consistent, and that they could use the equally-weighted mean of their two respective point estimates, $52.5$, as an overall estimate of the true mean.

The PhD student, however, argues that their two means should not be evenly weighted: she notes that her CI is half as wide and argues that her estimate is more precise and should thus be weighted more heavily. Her advisor notes that this cannot be correct, because the estimate from unevenly weighting the two means would be different from the estimate from analyzing the complete data set, which must be $52.5$. The PhD student's mistake is assuming that CIs directly indicate post-data precision.

The example above seems to be misleading. If we randomly divide a sample in half, into two samples, then we would expect both sample means and standard errors to be close. In such case there should not be any difference between using weighted mean (e.g. weighted by inverse errors) and using simple arithmetic mean. However if the estimates differ and errors in one of the samples is noticeably larger, this could suggest "issues" with such sample.

Obviously, in the above example, the sample sizes are the same so "joining back" the data by taking mean of the means is the same as taking mean of the whole sample. The problem is that the whole example follows the ill-defined logic that the sample is first divided in parts, then to be joined back again for the final estimate.

The example can be re-phrased to lead to exactly the opposite conclusion:

The researcher and the student decided to split their dataset in two halves and to analyze them independently. Afterwards, they compared their estimates and it appeared that the sample means that they calculated were very different, moreover standard error of student's estimate was much greater. The student was afraid that this could suggest issues with precision of his estimate, but the researcher implied that there is no connection between confidence intervals and precision, so both of the estimates are equally trustworthy and they can publish any one of them, chosen randomly, as their final estimate.

Stating it more formally, "standard" confidence intervals, like the Student's $t$, are based on errors

$$ \bar x \pm c \times \mathrm{SE}(x) $$

where $c$ is some constant. In such case, they are directly related to the precision, aren't they..?

So my question is:
Is the precision fallacy really a fallacy? What do confidence intervals say about precision?

Morey, R., Hoekstra, R., Rouder, J., Lee, M., & Wagenmakers, E.-J. (2015). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 1–21. https://learnbayes.org/papers/confidenceIntervalsFallacy/

  • 2
    $\begingroup$ I suppose if the precision's defined as the reciprocal of the variance, then the width of these CIs reflects only an estimate of the precision. Much as the width of a Bayesian credible interval for the mean would reflect uncertainty about the precision. $\endgroup$ Mar 30 '16 at 13:03
  • $\begingroup$ @Scortchi then this is another way of saying that the frequentist methods are unreliable in general..? $\endgroup$
    – Tim
    Mar 30 '16 at 13:08
  • 7
    $\begingroup$ I'd say this is a red herring. I just simulated 10,000 experiments, in each one drawing 50 samples from a normal distribution with mean 52.5 and SD 7.5 (so that the SE of subsamples of size 25 would be roughly $7.5/\sqrt{25}=5$, yielding CIs of $\pm 3$). I then split these samples in two and checked how often the CIs differed by 2 or more. This happened in only 6 out of 10,000 cases. Anyone observing CIs this different would much rather suspect something having broken in subsample selection. $\endgroup$ Mar 30 '16 at 13:08
  • $\begingroup$ @StephanKolassa I did exactly the same simulation that lead to exactly the same conclusions -- this is how the question emerged :) $\endgroup$
    – Tim
    Mar 30 '16 at 13:10
  • 2
    $\begingroup$ @Tim: I don't really know what they're trying to get at: if the true precision of the estimate of the mean is conceived of as a function of an unknown parameter value, common to the two sub-samples, then I don't think anyone would mind conceding that the difference in the width of these two CIs doesn't therefore reflect a difference in the precision of the estimates (unless they doubted the subsampling procedure). Considering the coverage properties of the CIs conditional on the observed coefficient of variation might have been a better tactic. $\endgroup$ Mar 30 '16 at 14:32

In the paper, we actually demonstrate the precision fallacy in multiple ways. The one you're asking about -- the first in the paper -- The example is meant to demonstrate that a simplistic "CI = precision" is wrong. This is not to say that any competent frequentist, Bayesian, or likelihoodist would be confused by this.

Here's another way to see what's going on: If we were just told the CIs, we would still not be able to combine the information in the samples together; we would need to know $N$, and from that we could decompose the CIs into the $\bar{x}$ and $s^2$, and thus combine the two samples properly. The reason we have to do this is that the information in the CI is marginal over the nuisance parameter. We must take into account that both samples contain information about the same nuisance parameter. This involves computing computing both $s^2$ values, combining them to get an overall estimate of $\sigma^2$, then computing a new CI.

As for other demonstrations of the precision fallacy, see

  • the multiple CIs in the Welch (1939) section (the submarine), one of which includes the "trivial" CI mentioned by @dsaxton above. In this example, the optimal CI does not track the width of the likelihood, and there are several other examples of CIs that do not either.
  • The fact that CIs -- even "good" CIs can be empty, "falsely" indicating infinite precision

The answer to the conundrum is that "precision", at least in the way CI advocates think about it (a post-experimental assessment of how "close" an estimate is to an parameter) is simply not a characteristic that confidence intervals have in general, and they were not meant to. Particular confidence procedures might...or not.

See also the discussion here: http://andrewgelman.com/2011/08/25/why_it_doesnt_m/#comment-61591

  • 7
    $\begingroup$ (+1) Great to hear from the actual author! I agree that CI's have several philosophical issues, as do ALL forms of inference (just different issues)...I like how you pointed out that it's the specific confidence procedure that you need to consider, not just that it is a CI at such and such level. $\endgroup$
    – user75138
    Mar 30 '16 at 18:37
  • 5
    $\begingroup$ (+1) Thanks for your response! I agree with arguments that you state in your paper that CI's do not have to say anything about precision, however calling this a fallacy gives impression that you indicate that they do not say anything about precision -- and this is not the same... Moreover: in your opinion, it the "precision fallacy" a real-life-analysis issue..? I agree that misinterpreting CI's is, but in this case, I'm not so sure... $\endgroup$
    – Tim
    Mar 30 '16 at 19:11
  • 2
    $\begingroup$ "Real-life" impact is difficult to quantify, particularly b/c one could talk about impact in a specific analysis scenario or across a field. For just computing a CI on a Gaussian mean the fallacy is not too dangerous. But consider the list of cites on p117 (para. starts "How often will Steiger’s confidence procedure..."). The intervals in those published papers is likely "too" narrow. The fallacy has other impacts: a lack of thoroughness on generators of new CI procedures (check any paper with a new CI), reluctance of analysts to move away from Gaussian assumptions when needed, and others. $\endgroup$ Mar 30 '16 at 20:19
  • $\begingroup$ I am tantalized by these parantheses. What is this "submarine"? $\endgroup$
    – Superbest
    Mar 30 '16 at 23:50
  • 1
    $\begingroup$ But if the width of the likelihood function for $\theta$ conditional on the sample range is supposed to truly reflect precision in the submarine example, then why shouldn't the width of the likelihood function for the mean conditional on the sample variance truly reflect the precision in this example. Suppose four bubbles from the submarine were observed & randomly split into two sets of two ... $\endgroup$ Mar 31 '16 at 10:46

First of all, lets limit ourselves to CI procedures that only produce intervals with strictly positive, finite widths (to avoid pathological cases).

In this case, the relationship between precision and CI width can be theoretically demonstrated. Take an estimate for the mean (when it exists). If your CI for the mean is very narrow, then you have two interpretations: either you had some bad luck and your sample was too tightly clumped (a priori 5% chance of that happening), or your interval covers the true mean (95% a priori chance). Of course, the observed CI can be either of these two, but, we set up our calculation so that the latter is far more likely to have occurred (i.e., 95% chance a priori)...hence, we have a high degree of confidence that our interval covers the mean, because we set things up probabilistically so this is so. Thus, a 95% CI is not a probability interval (like a Bayesian Credible Interval), but more like a "trusted adviser"...someone who, statistically, is right 95% of the time, so we trust their answers even though any particular answer could very well be wrong.

In the 95% of cases where it does cover the actual parameter, then the width tells you something about the range of plausible values given the data (i.e., how well you can bound the true value), hence it acts like a measure of precision. In the 5% of cases where it doesn't, then the CI is misleading (since the sample is misleading).

So, does 95% CI width indicate precision...I'd say there's a 95% chance it does (provided your CI width is positive-finite) ;-)

What is a sensible CI?

In response to the original author's post, I've revised my response to (a) take into account that the "split sample" example had a very specific purpose, and (b) to provide some more background as requested by the commenter:

In an ideal (frequentist) world, all sampling distributions would admit a pivotal statistic that we could use to get exact confidence intervals. What is so great about pivotal statistics? Their distribution can be derived without knowing the actual value of the parameter being estimated! In these nice cases, we have an exact distribution of our sample statistic relative to the true parameter (although it may not be gaussian) about this parameter.

Put more succinctly: We know the error distribution (or some transformation thereof).

It is this quality of some estimators that allows us to form sensible confidence intervals. These intervals don't just satisfy their definitions...they do so by virtue of being derived from the actual distribution of estimation error.

The Gaussian distribution and the associated Z statistic is the canonical example of the use of a pivotal quantity to develop an exact CI for the mean. There are more esoteric examples, but this is generally the one that motivates "large sample theory", which is basically an attempt apply the theory behind Gaussian CIs to distributions that do not admit a true pivotal quantity. In these cases, you'll read about approximately pivotal, or asymptotically pivotal (in the sample size) quantities or "approximate" confidence intervals...these are based on likelihood theory-- specifically, the fact that the error distribution for many MLEs approaches a normal distribution.

Another approach for generating sensible CIs is to "invert" a hypothesis test. The idea is that a "good" test (e.g., UMP) will result in a good (read: narrow) CI for a given Type I error rate. These don't tend to give exact coverage, but do provide lower-bound coverage (note: the actual definition of a X%-CI only says it must cover the true parameter at least X% of the time).

The use of hypothesis tests does not directly require a pivotal quantity or error distribution -- its sensibility is derived from the sensibility of the underlying test. For example, if we had a test whose rejection region had length 0 5% of the time and infinite length 95% of the time, we'd be back to where we were with the CI's -- but its obvious that this test is not conditional on the data, and hence will not provide any information on the underlying parameter being tested.

This broader idea - that an estimate of precision should be conditional on the data, goes back to Fischer and the idea of ancillary statistics. You can be sure that if the result of your test or CI procedure is NOT conditioned by the data (i.e., its conditional behavior is the same as its unconditional behavior), then you've got a questionable method on your hands.

  • 2
    $\begingroup$ It would be great if you could elaborate on what you added in a "Note". This is I think the crux of the whole discussion: one can devise very weird but valid frequentist procedures for constructing CIs under which the width of CI has no relationship to any precision whatsoever. Hence one can argue, as Morey et al. do, that CIs are misled in principle. I do agree with you that commonly used CI procedures are more reasonable than that, but one needs to be clear on what makes them such. $\endgroup$
    – amoeba
    Mar 30 '16 at 22:11
  • $\begingroup$ @amoeba I added some more explanation on why not all CIs are created equal...the main idea is ancillarity, the second is the role of an error distribution (or an approximation to it) $\endgroup$
    – user75138
    Mar 31 '16 at 3:02
  • $\begingroup$ Thanks for the update. One thing that I still don't find very clear in your answer, is that in the first paragraph you don't say anything about CI width; you are just talking about it containing or not containing the true population parameter. Everything there is correct even in "pathological" cases. Then in you say that yes, the width indicates precision, but you haven't provided any arguments for that (at that point). In the later discussion you explain it more though. $\endgroup$
    – amoeba
    Mar 31 '16 at 9:01
  • $\begingroup$ @amoeba I guess my post could do with a little more formatting. The basic logic is this (assuming we are using a "reasonable" CI procedure as I outline): there is an a priori 95% chance that the interval will contain the true parameter. After we collect data, we have our actual interval (finite, non-zero width). IF it contains the true parameter, then the width expresses the range of plausible values it could be, hence the width bounds the range of the parameter. HOWEVER, in the 5% of cases where the interval does not contain the value, then the interval is misleading. $\endgroup$
    – user75138
    Mar 31 '16 at 12:11
  • $\begingroup$ @amoeba updated post to better emphasize the connection between CI width and precision. $\endgroup$
    – user75138
    Mar 31 '16 at 12:19

I think the precision fallacy is a true fallacy, but not necessarily one we should care about. It isn't even that hard to show it's a fallacy. Take an extreme example like the following: we have a sample $\{x_1, x_2, \ldots , x_n \}$ from a normal$(\mu, \sigma^2)$ distribution and wish to construct a confidence interval on $\mu$, but instead of using the actual data we take our confidence interval to be either $(- \infty, \infty)$ or $\{ 0 \}$ based on the flip of a biased coin. By using the right bias we can get any level of confidence we like, but obviously our interval "estimate" has no precision at all even if we end up with an interval that has zero width.

The reason why I don't think we should care about this apparent fallacy is that while it is true that there's no necessary connection between the width of a confidence interval and precision, there is an almost universal connection between standard errors and precision, and in most cases the width of a confidence interval is proportional to a standard error.

I also don't believe the author's example is a very good one. Whenever we do data analysis we can only estimate precision, so of course the two individuals will reach different conclusions. But if we have some privileged knowledge, such as knowing that both samples are from the same distribution, then we obviously shouldn't ignore it. Clearly we should pool the data and use the resulting estimate of $\sigma$ as our best guess. It seems to me this example is like the one above where we only equate confidence interval width with precision if we've allowed ourselves to stop thinking.

  • $\begingroup$ Good point about the randomly infinite CIs...definitely show that confidence is a different concept than precision. I probably should have caveated my response by saying that I am assuming a likelihood-based CI, where width is related to curvature of log likelihood, which is an approximation of standard error...your post points out that there are CIs that technically achieve coverage but in a very counterintuitive way. $\endgroup$
    – user75138
    Mar 30 '16 at 14:37
  • $\begingroup$ A related issue (albeit very interesting one) is that of relevant subsets for a CI...for example, if you condition on ancillary statistics, your CI coverage may change (a case in point is that the conditional coverage of a t-interval changes based on the variability of your sample). Here's the link to the paper: jstor.org/stable/2242024?seq=1#page_scan_tab_contents $\endgroup$
    – user75138
    Mar 30 '16 at 14:38
  • $\begingroup$ @Bey There's another less extreme example from this paper involving a submarine: webfiles.uci.edu/mdlee/fundamentalError.pdf. It's an interesting one, but again appears to be a case of an interpretation that no intelligent person would make. $\endgroup$
    – dsaxton
    Mar 30 '16 at 14:48
  • $\begingroup$ Agreed....can't leave common sense at the door with stats...even in Machine Learning (somewhat of a misnomer) $\endgroup$
    – user75138
    Mar 30 '16 at 16:13
  • 1
    $\begingroup$ @richarddmorey: Okay, I see. Then it was just an unfortunate formulation! I did not take it out of the context on purpose; I honestly read this sentence as a summary and generalization to any situation (not realizing that "in that example" was assumed in that sentence). Consider leaving a clarification comment in that other thread with my accusation (that already got some upvotes). $\endgroup$
    – amoeba
    Mar 31 '16 at 13:22

I think the demonstrable distinction between "confidence intervals" and "precision" (see answer from @dsaxton) is important because that distinction points out problems in common usage of both terms.

Quoting from Wikipedia:

The precision of a measurement system, related to reproducibility and repeatability, is the degree to which repeated measurements under unchanged conditions show the same results.

One thus might argue that frequentist confidence intervals do represent a type of precision of a measurement scheme. If one repeats the same scheme, the 95% CI calculated for each repetition will contain the one true value of the parameter in 95% of the repetitions.

This, however, is not what many people want from a practical measure of precision. They want to know how close the measured value is to the true value. Frequentist confidence intervals do not strictly provide that measure of precision. Bayesian credible regions do.

Some of the confusion is that, in practical examples, frequentist confidence intervals and Bayesian credible regions "will more-or-less overlap". Sampling from a normal distribution, as in some comments on the OP, is such an example. That may also be the case in practice for some of the broader types of analyses that @Bey had in mind, based on approximations to standard errors in processes that have normal distributions in the limit.

If you know that you are in such a situation, then there may be no practical danger in interpreting a particular 95% CI, from a single implementation of a measurement scheme, as having a 95% probability of containing the true value. That interpretation of confidence intervals, however, is not from frequentist statistics, for which the true value either is or is not within that particular interval.

If confidence intervals and credible regions differ markedly, that Bayesian-like interpretation of frequentist confidence intervals can be misleading or wrong, as the paper linked above and earlier literature referenced therein demonstrate. Yes, "common sense" might help avoid such misinterpretations, but in my experience "common sense" isn't so common.

Other CrossValidated pages contain much more information on confidence intervals and the differences between confidence intervals and credible regions. Links from those particular pages are also highly informative.

  • $\begingroup$ This is a good point....I think the closest think to the common interpretation of "precision" is more like RMS error. An unbiased but highly variable estimate is seen as no better than a low-variability but highly biased estimator...both cannot be relied upon to give an estimate close to the true value. $\endgroup$
    – user75138
    Mar 30 '16 at 17:58
  • $\begingroup$ +1, but I am not sure I share your pessimistic view on "common sense". There is a great quote from Jeffreys about "common sense" in frequentist statistics: I have in fact been struck repeatedly in my own work, after being led on general principles to the solution of a problem, to find that Fisher had already grasped the essentials by some brilliant piece of common sense. $\endgroup$
    – amoeba
    Mar 30 '16 at 22:05
  • $\begingroup$ @amoeba consider Laplace's claim that "Probability theory is nothing but common sense reduced to calculation." The efforts devoted since then to probability theory at least show that the implications of common sense aren't always immediately obvious. $\endgroup$
    – EdM
    Mar 30 '16 at 22:44
  • $\begingroup$ @amoeba: Fisher rejected CIs, and identifying Fisher as freq-ist. is misleading. His logic of intervals (fiducial) was similar to obj. Bayes, and he identifies probability with rational uncertainty. He says this: "It is sometimes asserted that the fiducial method generally leads to the same results as the method of [CIs]. It is difficult to understand how this can be so, since it has been firmly laid down that the method of confidence intervals does not lead to probability statements about the parameters of the real world, whereas the fiducial argument exists for this purpose." (Fisher, 1959) $\endgroup$ Mar 31 '16 at 11:37
  • $\begingroup$ @richard, Thanks for the clarification. Fisher is known to have said contradictory things throughout his long career and to have changed his opinion a couple of times. I am not really familiar with his fiducial theory so cannot comment on that. My unconscious assumption was that Jeffreys in that quote was referring to the Fisher's "frequentist period" but I have no evidence for that. In my (limited!) experience, nobody ever uses fiducial inference. Nobody. Ever. Whereas frequentist techniques are used all the time and many go back to Fisher. Hence the association existing in my mind. $\endgroup$
    – amoeba
    Mar 31 '16 at 11:43

@Bey has it. There is no necessary connection between scores and performance nor price and quality nor smell and taste. Yet the one usually informs about the other.

One can prove by induction that one cannot give a pop quiz. On close examination this means one cannot guarantee the quiz is a surprise. Yet most of the time it will be.

It sounds like Morey et al show that there exist cases where the width is uninformative. Although that is sufficient to claim "There is no necessary connection between the precision of an estimate and the size of a confidence interval", it is not sufficient to further conclude that CIs generally contain no information about precision. Merely that they are not guaranteed to do so.

(Insufficient points to + @Bey's answer. )


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.