I was reading a blog post by the statistician William Briggs, and the following claim interested me to say the least.

What do you make of it?

What is a confidence interval? It is an equation, of course, that will provide you an interval for your data. It is meant to provide a measure of the uncertainty of a parameter estimate. Now, strictly according to frequentist theory—which we can even assume is true—the only thing you can say about the CI you have in hand is that the true value of the parameter lies within it or that it does not. This is a tautology, therefore it is always true. Thus, the CI provides no measure of uncertainty at all: in fact, it is a useless exercise to compute one.

Link: http://wmbriggs.com/post/3169/

  • 3
    $\begingroup$ Without a precise reference, there is, most crucially, no context here. There is also no way to get indications of the style and credentials of William Briggs (not known to me). It could be that here is someone who just likes to be provocative and outrageous. There are, naturally, deep and difficult technical and philosophical issues here too, which are the question, but asking us to debate a quotation with no background is (one view only) unlikely to be fruitful. $\endgroup$ – Nick Cox Feb 20 '15 at 16:21
  • 1
    $\begingroup$ @NickCox With regards to the omission of relevant context, I have now edited the initial post. $\endgroup$ – Five σ Feb 20 '15 at 16:22
  • 6
    $\begingroup$ Thanks very much for providing the back-up. It's only a comment and I lack the inclination to extend it, but my three-word reaction is that the last sentence is an exaggerated claim. You can hope for much fuller answers. $\endgroup$ – Nick Cox Feb 20 '15 at 16:28
  • 1
    $\begingroup$ @NickCox No problem Nick. However, I do appreciate your sentiments as it was sloppy of me to not reference my question. $\endgroup$ – Five σ Feb 20 '15 at 16:28
  • 2
    $\begingroup$ @Nick I would say Briggs succeeded in one of his two objectives: "Today’s thoughts are just a sketch to help clear my mind and begin a discussion. Meaning, it’s likely I will have fallen prey to my own complaint" (that your "neighborhood statistician" is a "sloppy thinker"). $\endgroup$ – whuber Feb 20 '15 at 17:46

He's referring, rather clumsily, to the well known fact that frequentist analysis doesn't model the state of our knowledge about an unknown parameter with a probability distribution, so having calculated a (say 95%) confidence interval (say 1.2 to 3.4) for a population parameter (say the mean of a Gaussian distribution) from some data you can't then go ahead & claim that there's a 95% probability of the mean falling between 1.2 and 3.4. The probability's one or zero—you don't know which. But what you can say, in general, is that your procedure for calculating 95% confidence intervals is one that ensures they contain the true parameter value 95% of the time. This seems reason enough for saying that CIs reflect uncertainty. As Sir David Cox put it

We define procedures for assessing evidence that are calibrated by how they would perform were they used repeatedly. In that sense they do not differ from other measuring instruments.

See here & here for further explanation.

Other things you can say vary according to the particular method you used to calculate the confidence interval; if you ensure the values inside have greater likelihood, given the data, than the points outside, then you can say that (& it's often approximately true for commonly used methods). See here for more.

† Cox (2006), Principles of Statistical Inference, §1.5.2

  • 1
    $\begingroup$ That's Sir David Cox, I imagine. $\endgroup$ – Nick Cox Feb 20 '15 at 18:22
  • $\begingroup$ @NickCox: It is indeed. $\endgroup$ – Scortchi Feb 21 '15 at 21:44
  • $\begingroup$ Is Sir David's quoted analogy correct? (Not a correct quote, but a correct analogy.) I don't imagine a thermometer which 95% of the time reports the temperature $\pm\epsilon$, but 5% of the time reports temperature outside of $\pm\epsilon$ -- and perhaps far outside of that range? $\endgroup$ – Wayne Feb 21 '15 at 22:22
  • 1
    $\begingroup$ @Spectrosaurus: The posts I linked to go into this in more detail. In fine, the population mean $\mu$ isn't modelled as a random variable; the data $\vec{X}_\mu$ are, with a distribution that depends on $\mu$, & the confidence interval $(b_\mathrm{L}(X_\mu), b_\mathrm{U}(X_\mu))$ is a function of the data. $\Pr[b_\mathrm{U}(\vec{X}_\mu) < \mu < b_\mathrm{U}(\vec{X}_\mu)]=0.95$ defines a valid confidence interval with 95% coverage, whatever value $\mu$ might have. So if $\mu=2$, ... $\endgroup$ – Scortchi Jun 19 at 14:13
  • 1
    $\begingroup$ ... $\Pr[b_\mathrm{U}(\vec{X}_2) < 2 < b_\mathrm{U}(\vec{X}_2)]=0.95$ is true, & if $\mu=7$, $\Pr[b_\mathrm{U}(\vec{X}_7) < 7 < b_\mathrm{U}(\vec{X}_7)]=0.95$ is true. Now substituting in realized values of $X_\mu$ gives e.g. $\Pr[1.2 < \mu < 3.4]=0.95$, i.e. if $\mu=2$, $\Pr[1.2 < 2 < 3.4]=0.95$ & if $\mu=7$, $\Pr[1.2 < 2 < 3.4]=0.95$ - which is nonsense. $\endgroup$ – Scortchi Jun 19 at 14:13

It can be hard to mathematically characterize uncertainty, but I know it when I see it; it usually has wide 95% confidence intervals.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.