What do confidence intervals say about precision (if anything)?

Question

Morey et al (2015) argue that confidence intervals are misleading and there are multiple bias related to understanding of them. Among others, they describe the precision fallacy as following:

The Precision fallacy
The width of a confidence interval indicates the precision of our knowledge about the parameter. Narrow confidence intervals show precise knowledge, while wide confidence errors show imprecise knowledge.

There is no necessary connection between the precision of an estimate and the size of a confidence interval. One way to see this is to imagine two researchers — a senior researcher and a PhD student — are analyzing data of $50$ participants from an experiment. As an exercise for the PhD student's benefit, the senior researcher decides to randomly divide the participants into two sets of $25$ so that they can each separately analyze half the data set. In a subsequent meeting, the two share with one another their Student's $t$ confidence intervals for the mean. The PhD student's $95\%$ CI is $52 \pm 2$, and the senior researcher's $95\%$ CI is $53 \pm 4$.

The senior researcher notes that their results are broadly consistent, and that they could use the equally-weighted mean of their two respective point estimates, $52.5$, as an overall estimate of the true mean.

The PhD student, however, argues that their two means should not be evenly weighted: she notes that her CI is half as wide and argues that her estimate is more precise and should thus be weighted more heavily. Her advisor notes that this cannot be correct, because the estimate from unevenly weighting the two means would be different from the estimate from analyzing the complete data set, which must be $52.5$. The PhD student's mistake is assuming that CIs directly indicate post-data precision.

The example above seems to be misleading. If we randomly divide a sample in half, into two samples, then we would expect both sample means and standard errors to be close. In such case there should not be any difference between using weighted mean (e.g. weighted by inverse errors) and using simple arithmetic mean. However if the estimates differ and errors in one of the samples is noticeably larger, this could suggest "issues" with such sample.

Obviously, in the above example, the sample sizes are the same so "joining back" the data by taking mean of the means is the same as taking mean of the whole sample. The problem is that the whole example follows the ill-defined logic that the sample is first divided in parts, then to be joined back again for the final estimate.

The example can be re-phrased to lead to exactly the opposite conclusion:

The researcher and the student decided to split their dataset in two halves and to analyze them independently. Afterwards, they compared their estimates and it appeared that the sample means that they calculated were very different, moreover standard error of student's estimate was much greater. The student was afraid that this could suggest issues with precision of his estimate, but the researcher implied that there is no connection between confidence intervals and precision, so both of the estimates are equally trustworthy and they can publish any one of them, chosen randomly, as their final estimate.

Stating it more formally, "standard" confidence intervals, like the Student's $t$, are based on errors

$$ \bar x \pm c \times \mathrm{SE}(x) $$

where $c$ is some constant. In such case, they are directly related to the precision, aren't they..?

So my question is:
Is the precision fallacy really a fallacy? What do confidence intervals say about precision?

---

Morey, R., Hoekstra, R., Rouder, J., Lee, M., & Wagenmakers, E.-J. (2015). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 1–21. https://learnbayes.org/papers/confidenceIntervalsFallacy/

Answer 1

In the paper, we actually demonstrate the precision fallacy in multiple ways. The one you're asking about --- the first in the paper --- The example is meant to demonstrate that a simplistic "CI = precision" is wrong. This is not to say that any competent frequentist, Bayesian, or likelihoodist would be confused by this.

Here's another way to see what's going on: If we were just told the CIs, we would still not be able to combine the information in the samples together; we would need to know $N$, and from that we could decompose the CIs into the $\bar{x}$ and $s^2$, and thus combine the two samples properly. The reason we have to do this is that the information in the CI is marginal over the nuisance parameter. We must take into account that both samples contain information about the same nuisance parameter. This involves computing both $s^2$ values, combining them to get an overall estimate of $\sigma^2$, then computing a new CI.

As for other demonstrations of the precision fallacy, see

• the multiple CIs in the Welch (1939) section (the submarine), one of which includes the "trivial" CI mentioned by @dsaxton above. In this example, the optimal CI does not track the width of the likelihood, and there are several other examples of CIs that do not either.
• The fact that CIs --- even "good" CIs can be empty, "falsely" indicating infinite precision

The answer to the conundrum is that "precision", at least in the way CI advocates think about it (a post-experimental assessment of how "close" an estimate is to an parameter) is simply not a characteristic that confidence intervals have in general, and they were not meant to. Particular confidence procedures might ... or not.

See also the discussion here: http://andrewgelman.com/2011/08/25/why_it_doesnt_m/#comment-61591

Answer 2

First of all, lets limit ourselves to CI procedures that only produce intervals with strictly positive, finite widths (to avoid pathological cases).

In this case, the relationship between precision and CI width can be theoretically demonstrated. Take an estimate for the mean (when it exists). If your CI for the mean is very narrow, then you have two interpretations: either you had some bad luck and your sample was too tightly clumped (a priori 5% chance of that happening), or your interval covers the true mean (95% a priori chance). Of course, the observed CI can be either of these two, but, we set up our calculation so that the latter is far more likely to have occurred (i.e., 95% chance a priori)...hence, we have a high degree of confidence that our interval covers the mean, because we set things up probabilistically so this is so. Thus, a 95% CI is not a probability interval (like a Bayesian Credible Interval), but more like a "trusted adviser"...someone who, statistically, is right 95% of the time, so we trust their answers even though any particular answer could very well be wrong.

In the 95% of cases where it does cover the actual parameter, then the width tells you something about the range of plausible values given the data (i.e., how well you can bound the true value), hence it acts like a measure of precision. In the 5% of cases where it doesn't, then the CI is misleading (since the sample is misleading).

So, does 95% CI width indicate precision...I'd say there's a 95% chance it does (provided your CI width is positive-finite) ;-)

What is a sensible CI?

In response to the original author's post, I've revised my response to (a) take into account that the "split sample" example had a very specific purpose, and (b) to provide some more background as requested by the commenter:

In an ideal (frequentist) world, all sampling distributions would admit a pivotal statistic that we could use to get exact confidence intervals. What is so great about pivotal statistics? Their distribution can be derived without knowing the actual value of the parameter being estimated! In these nice cases, we have an exact distribution of our sample statistic relative to the true parameter (although it may not be gaussian) about this parameter.

Put more succinctly: We know the error distribution (or some transformation thereof).

It is this quality of some estimators that allows us to form sensible confidence intervals. These intervals don't just satisfy their definitions...they do so by virtue of being derived from the actual distribution of estimation error.

The Gaussian distribution and the associated Z statistic is the canonical example of the use of a pivotal quantity to develop an exact CI for the mean. There are more esoteric examples, but this is generally the one that motivates "large sample theory", which is basically an attempt apply the theory behind Gaussian CIs to distributions that do not admit a true pivotal quantity. In these cases, you'll read about approximately pivotal, or asymptotically pivotal (in the sample size) quantities or "approximate" confidence intervals...these are based on likelihood theory-- specifically, the fact that the error distribution for many MLEs approaches a normal distribution.

Another approach for generating sensible CIs is to "invert" a hypothesis test. The idea is that a "good" test (e.g., UMP) will result in a good (read: narrow) CI for a given Type I error rate. These don't tend to give exact coverage, but do provide lower-bound coverage (note: the actual definition of a X%-CI only says it must cover the true parameter at least X% of the time).

The use of hypothesis tests does not directly require a pivotal quantity or error distribution -- its sensibility is derived from the sensibility of the underlying test. For example, if we had a test whose rejection region had length 0 5% of the time and infinite length 95% of the time, we'd be back to where we were with the CI's -- but its obvious that this test is not conditional on the data, and hence will not provide any information on the underlying parameter being tested.

This broader idea - that an estimate of precision should be conditional on the data, goes back to Fischer and the idea of ancillary statistics. You can be sure that if the result of your test or CI procedure is NOT conditioned by the data (i.e., its conditional behavior is the same as its unconditional behavior), then you've got a questionable method on your hands.

Answer 3

I think the precision fallacy is a true fallacy, but not necessarily one we should care about. It isn't even that hard to show it's a fallacy. Take an extreme example like the following: we have a sample $\{x_1, x_2, \ldots , x_n \}$ from a normal$(\mu, \sigma^2)$ distribution and wish to construct a confidence interval on $\mu$, but instead of using the actual data we take our confidence interval to be either $(- \infty, \infty)$ or $\{ 0 \}$ based on the flip of a biased coin. By using the right bias we can get any level of confidence we like, but obviously our interval "estimate" has no precision at all even if we end up with an interval that has zero width.

The reason why I don't think we should care about this apparent fallacy is that while it is true that there's no necessary connection between the width of a confidence interval and precision, there is an almost universal connection between standard errors and precision, and in most cases the width of a confidence interval is proportional to a standard error.

I also don't believe the author's example is a very good one. Whenever we do data analysis we can only estimate precision, so of course the two individuals will reach different conclusions. But if we have some privileged knowledge, such as knowing that both samples are from the same distribution, then we obviously shouldn't ignore it. Clearly we should pool the data and use the resulting estimate of $\sigma$ as our best guess. It seems to me this example is like the one above where we only equate confidence interval width with precision if we've allowed ourselves to stop thinking.

Answer 4

I think the demonstrable distinction between "confidence intervals" and "precision" (see answer from @dsaxton) is important because that distinction points out problems in common usage of both terms.

Quoting from Wikipedia:

The precision of a measurement system, related to reproducibility and repeatability, is the degree to which repeated measurements under unchanged conditions show the same results.

One thus might argue that frequentist confidence intervals do represent a type of precision of a measurement scheme. If one repeats the same scheme, the 95% CI calculated for each repetition will contain the one true value of the parameter in 95% of the repetitions.

This, however, is not what many people want from a practical measure of precision. They want to know how close the measured value is to the true value. Frequentist confidence intervals do not strictly provide that measure of precision. Bayesian credible regions do.

Some of the confusion is that, in practical examples, frequentist confidence intervals and Bayesian credible regions "will more-or-less overlap". Sampling from a normal distribution, as in some comments on the OP, is such an example. That may also be the case in practice for some of the broader types of analyses that @Bey had in mind, based on approximations to standard errors in processes that have normal distributions in the limit.

If you know that you are in such a situation, then there may be no practical danger in interpreting a particular 95% CI, from a single implementation of a measurement scheme, as having a 95% probability of containing the true value. That interpretation of confidence intervals, however, is not from frequentist statistics, for which the true value either is or is not within that particular interval.

If confidence intervals and credible regions differ markedly, that Bayesian-like interpretation of frequentist confidence intervals can be misleading or wrong, as the paper linked above and earlier literature referenced therein demonstrate. Yes, "common sense" might help avoid such misinterpretations, but in my experience "common sense" isn't so common.

Other CrossValidated pages contain much more information on confidence intervals and the differences between confidence intervals and credible regions. Links from those particular pages are also highly informative.

Answer 5

@Bey has it. There is no necessary connection between scores and performance nor price and quality nor smell and taste. Yet the one usually informs about the other.

One can prove by induction that one cannot give a pop quiz. On close examination this means one cannot guarantee the quiz is a surprise. Yet most of the time it will be.

It sounds like Morey et al show that there exist cases where the width is uninformative. Although that is sufficient to claim "There is no necessary connection between the precision of an estimate and the size of a confidence interval", it is not sufficient to further conclude that CIs generally contain no information about precision. Merely that they are not guaranteed to do so.

(Insufficient points to + @Bey's answer. )