Understanding confidence intervals - how can there be several ways to compute a CI?

Question

I'm struggling with understanding confidence intervals and the common fallacies that I'm obviously not alone in doing. For example, why can't I say that I'm 95% confidence that the true value lies within my 95% confidence interval if I've made a measurement with a device that has a normal distributed error with known standard deviation?

In an attempt to understand this better I read the submarine example in the article The Fallacy of Placing Confidence in Confidence Intervals by Morey, R.D., Hoekstra, R., Lee, M.D., Rouder, J.N., Wagenmakers, E-J, the example goes like this:

"A 10-meter-long research submersible with several people on board has lost contact with its surface support vessel. The submersible has a rescue hatch exactly halfway along its length, to which the support vessel will drop a rescue line. Because the rescuers only get one rescue attempt, it is crucial that when the line is dropped to the craft in the deep water that the line be as close as possible to this hatch. The researchers on the support vessel do not know where the submersible is, but they do know that it forms two dis- tinctive bubbles. These bubbles could form anywhere along the craft’s length, independently, with equal probability, and float to the surface where they can be seen by the support vessel."

They go on by showing several different ways of calculating a 50% confidence interval which highlights the fallacies of confidence and precision very well. However I can't really translate the example into a real-world scenario. Say that I'm performing a measurement with a device that has a normal distributed error with known standard deviation. Are there other ways to calculate the 95% confidence interval besides $[x-1.96\sigma, x+1.96\sigma]$ in that scenario as well?

Answer 1

For your normal distribution example $(-\infty, \infty)$ will cover the true value $\geq$ 95% of the time. So will $(x-2.326 \sigma, x+1.751\sigma)$, so will many other possible intervals. Whether these are sensible intervals that you would use in practice is another question.

That's the issue with the submarine example to me: a lot of the frequentist confidence intervals they show are ones that I hope no sensible frequentist would ever propose. I suppose you might say that they fail to make a point that is relevant in practice by emphasizing intervals that technically are valid 95% confidence intervals, but are otherwise rather idiotic. Something like a minimum length confidence interval based on a sensible model likelihood will give something rather more sensible.

These points are sort of totally unrelated to the first point regarding the interpretation of confidence intervals. They will cover the true value 95% of the time, if you repeatedly perform the same experiment and your model is right. However, on each iteration either the true value is covered by this random interval or not, if you consider the interval as a random variable and the true parameter as a fixed quantity. It has always seemed to me that making too much of a distinction between a Bayesian credible interval with (close to) non-informative priors and such a confidence interval is rather irrelevant in practice and claiming that this somehow allows a totally different interpretation often seems a bit like sophistry. However, it starts to make a lot more sense when we are talking about Bayesian credible intervals that include (weakly) informative prior information that is available.

Answer 2

Q: How can there be several ways to compute a CI?

A: Because there are several different ways to estimate the value of a parameter and your choice of estimation technique has limitations that will reflect the size of the corresponding CI.

The choice of modeling determines what you can say about the estimated parameters. If you check out the Wikipedia article on confidence intervals you'll see that the way you calculate a confidence interval from likelihood modeling is different from how you calculate CIs when you're using a general linear model. Likelihood methods usually employ two different ways of calculating a CI; one method looks like you're used to from general linear modeling with the estimated value and a $\pm$ some percentage of a distribution above and below the estimated value. In order to do that the likelihood function needs to be regular. In case the likelihood function is not regular you'll have to use likelihood ratios and relate the cutoff value to a $\chi^2$ distribution. (roughly speaking, I'm trying to illustrate the different limitations of the technique).

In other words: the way you analyze your data, determines what you can say about your estimates. Your ability to say something about your estimates, depend on how you actually estimate your parameters. Notice that the data itself does not determine the size of your CI, but the proper model for that data type has particular properties that determine what the CI will look like. Some methods have analytical expressions for how to calculate CIs, many methods do not.

What that in mind, consider why bootstrapping is so nice.

So to refer to the last paragraph in your question:

Are there other ways to calculate the 95% confidence interval besides $[x−1.96σ,x+1.96σ]$ in that scenario as well?

Yes. By using other models on the same data set. To give a visual explanation (and to link to an interesting video) Check out the youtube channel Veritasium and the video: Is Most Published Research Wrong?. I've linked to 8:12 into the video. Just pause it and take a look at the graph. 29 ways to analyze the same data set comes up with vastly different confidence intervals.

All this to say: A confidence interval says something about your model as well as saying something about your parameters estimates.