I'm struggling with understanding confidence intervals and the common fallacies that I'm obviously not alone in doing. For example, why can't I say that I'm 95% confidence that the true value lies within my 95% confidence interval if I've made a measurement with a device that has a normal distributed error with known standard deviation?

In an attempt to understand this better I read the submarine example in the article The Fallacy of Placing Confidence in Confidence Intervals by Morey, R.D., Hoekstra, R., Lee, M.D., Rouder, J.N., Wagenmakers, E-J, the example goes like this:

"A 10-meter-long research submersible with several people on board has lost contact with its surface support vessel. The submersible has a rescue hatch exactly halfway along its length, to which the support vessel will drop a rescue line. Because the rescuers only get one rescue attempt, it is crucial that when the line is dropped to the craft in the deep water that the line be as close as possible to this hatch. The researchers on the support vessel do not know where the submersible is, but they do know that it forms two dis- tinctive bubbles. These bubbles could form anywhere along the craft’s length, independently, with equal probability, and float to the surface where they can be seen by the support vessel."

They go on by showing several different ways of calculating a 50% confidence interval which highlights the fallacies of confidence and precision very well. However I can't really translate the example into a real-world scenario. Say that I'm performing a measurement with a device that has a normal distributed error with known standard deviation. Are there other ways to calculate the 95% confidence interval besides $[x-1.96\sigma, x+1.96\sigma]$ in that scenario as well?

  • $\begingroup$ In your toy example, $(-\infty, x+1.645\sigma]$ or $[x-2.326\sigma, x+ 1.751\sigma]$ are $95\%$ confidence intervals. So too is choosing $(-\infty, +\infty)$ with probability $0.95$ and $[x,x]$ with probability $0.05$ $\endgroup$ – Henry Sep 15 '16 at 7:28
  • $\begingroup$ Good point! So in your first example that is the CI used if it makes sense to make a one-sided CI, right? But whether to use a two- or one-sided CI should be determined by what kind of measurement we're doing right? Is it common to be in a situation where it's not clear whether to use a one-sided or a two-sided CI? $\endgroup$ – Alex Sep 15 '16 at 7:58

For your normal distribution example $(-\infty, \infty)$ will cover the true value $\geq$ 95% of the time. So will $(x-2.326 \sigma, x+1.751\sigma)$, so will many other possible intervals. Whether these are sensible intervals that you would use in practice is another question.

That's the issue with the submarine example to me: a lot of the frequentist confidence intervals they show are ones that I hope no sensible frequentist would ever propose. I suppose you might say that they fail to make a point that is relevant in practice by emphasizing intervals that technically are valid 95% confidence intervals, but are otherwise rather idiotic. Something like a minimum length confidence interval based on a sensible model likelihood will give something rather more sensible.

These points are sort of totally unrelated to the first point regarding the interpretation of confidence intervals. They will cover the true value 95% of the time, if you repeatedly perform the same experiment and your model is right. However, on each iteration either the true value is covered by this random interval or not, if you consider the interval as a random variable and the true parameter as a fixed quantity. It has always seemed to me that making too much of a distinction between a Bayesian credible interval with (close to) non-informative priors and such a confidence interval is rather irrelevant in practice and claiming that this somehow allows a totally different interpretation often seems a bit like sophistry. However, it starts to make a lot more sense when we are talking about Bayesian credible intervals that include (weakly) informative prior information that is available.

| cite | improve this answer | |
  • $\begingroup$ When you say that it's "irrelevant in practice", do you mean that it does make sense to say "I'm 95% confident that the true value is within this CI"? If the CI will cover the true value 95% of the times it sounds like a reasonable statement to me... (I added +1 on your answer but my rep is too low for it to show, but thanks!) $\endgroup$ – Alex Sep 15 '16 at 7:46
  • $\begingroup$ Actually, I would not say that the I have 95% probability that the value is within the range of the 95% confidence (or credible interval with uninformative priors). I am not sure whether that changes at all, if I use the word "confident" (with some technical explanation of what that's meant to mean in the background - certainly not if it just means "probable"). When I have good/relevant and really informative prior information and have a 95% credible interval, this really makes sense (and it starts to make sense to claim a difference in interpretation versus a confidence interval). $\endgroup$ – Björn Sep 15 '16 at 10:14

Q: How can there be several ways to compute a CI?

A: Because there are several different ways to estimate the value of a parameter and your choice of estimation technique has limitations that will reflect the size of the corresponding CI.

The choice of modeling determines what you can say about the estimated parameters. If you check out the Wikipedia article on confidence intervals you'll see that the way you calculate a confidence interval from likelihood modeling is different from how you calculate CIs when you're using a general linear model. Likelihood methods usually employ two different ways of calculating a CI; one method looks like you're used to from general linear modeling with the estimated value and a $\pm$ some percentage of a distribution above and below the estimated value. In order to do that the likelihood function needs to be regular. In case the likelihood function is not regular you'll have to use likelihood ratios and relate the cutoff value to a $\chi^2$ distribution. (roughly speaking, I'm trying to illustrate the different limitations of the technique).

In other words: the way you analyze your data, determines what you can say about your estimates. Your ability to say something about your estimates, depend on how you actually estimate your parameters. Notice that the data itself does not determine the size of your CI, but the proper model for that data type has particular properties that determine what the CI will look like. Some methods have analytical expressions for how to calculate CIs, many methods do not.

What that in mind, consider why bootstrapping is so nice.

So to refer to the last paragraph in your question:

Are there other ways to calculate the 95% confidence interval besides $[x−1.96σ,x+1.96σ]$ in that scenario as well?

Yes. By using other models on the same data set. To give a visual explanation (and to link to an interesting video) Check out the youtube channel Veritasium and the video: Is Most Published Research Wrong?. I've linked to 8:12 into the video. Just pause it and take a look at the graph. 29 ways to analyze the same data set comes up with vastly different confidence intervals.

All this to say: A confidence interval says something about your model as well as saying something about your parameters estimates.

| cite | improve this answer | |
  • $\begingroup$ And I assume there is no "correct" method for parameter estimation to choose from in each case? Looking at the example in the linked video my first thought is that some research groups has made a poor choice of method to estimate the true value, but you're sayinng they all could be valid methods depending on ...what exactly? I understand that assumptions about the underlying distribution will affect the CI but isn't there some sort of "correct" assumption of the underlying distribution tht can be made in each case? $\endgroup$ – Alex Sep 15 '16 at 9:47
  • 1
    $\begingroup$ There's plenty of cases where there's plenty of reasonable confidence intervals (e.g. from an analysis of covariance adjusted for covariate X1, adjusted for covariates X1, X2 and X3, adjusted for X2, X3 and X4 etc.). Any of them may be reasonable and the biggest concern is that people would not pre-specify exactly what they will do and then pick the one that gives them the result they would like to see (or change the definition of the outcome variable or look at different outcomes variables until they get what they want). Then the interval will not cover the true value in 95% of studies. $\endgroup$ – Björn Sep 15 '16 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.