95% Confidence Intervals

Question

Statistics textbooks go out of their way to say that 95% Confidence Intervals (CIs) do not mean that you can be 95% sure that the population parameter of interest is somewhere between the high and low end of the interval. Rather, if your sample was drawn an infinite number of times, 95% of the intervals would contain the population parameter (while 5% would not).

I fail to see the distinction. If I draw one of the infinite number of samples for which 95% CIs are calculated, aren’t I 95% certain that I’ve drawn one the ones whose CI contains the population parameter? Thus, I’m 95% certain that my CI contains the population parameter.

If someone can explain why my thinking is incorrect, I’d really appreciate it. Thank you.

---

Just to cause more confusion, I went to my old Statistical Methods textbook by Snedecor and Cochran (8th edition), and found the following section on Confidence Intervals:

Notice that they provide a mathematical proof for the inequality relating a population parameter value to a sample confidence interval. In addition, in their example in the middle of page 56, they explicitly state that the population parameter lies within the 95% confidence interval given, except in a 1 in 20 chance.

Snedecor and Cochran's book educated several generations of statisticians, at least here in the US. And, the mathematical proof seems pretty convincing. So now what? Do we believe what the current textbooks are saying (which do not help us in making a statement about the population parameter)? Or, do we go with Snedecor and Cochran and state that we are 95% certain the the population parameter is within our 95% CIs?

Anyone who wishes to comment, please do...I'm at a loss.

Answer 1

The clue to all of this is realizing that the population paramter $\theta$ is a fixed, unknown number. And that (loosely speaking) the "randomnes" in all of this comes from the confidence intervals. Each confidence interval is linked to a sample, so for different samples, we get (ideally slightly) different CIs

Now, given a population $X$, consider a simple random sample (SRS) of size $n$ $\underline{X}_n=(X_1,X_2,\ldots,X_n)$ that depend on the unknown parameter $\theta$.

A confidence interval estimator for $\theta$ at a confidece level of $95\%$ is an interval $(T_1(\underline{X}_n), T_2(\underline{X}_n))$ that satisfies that $$P(\theta\in (T_1(\underline{X}_n), T_2(\underline{X}_n)) = 95\%$$

Now $\underline{X}_n$ was a SRS so for this SRS I obtain a specific sample $\underline{x}_n=(x_1,x_2,\ldots x_n)$. While $\underline{X}_n$ was a bunch of random variables, $\underline{x}_n$ is a bunch of specific numbers. So I use this specific sample and I obtain one specific confidence interval linked to this sample $CI(\theta)_{95}=(a,b)$ where now $a\in\mathbb{R}$ and $b\in\mathbb{R}$.

Taking into account that $\theta$ is a fixed number, there are two possible results. Or $\theta$ is inside this CI or it is outside this CI:

1. If $\theta\in(a,b)$ then in this case $P(\theta\in(a,b))=1$
2. If $\theta\notin(a,b)$. then in this case $P(\theta\in(a,b))=0$

EDIT adding example

In the end, it is simply a problem of language. Consider that the parameter under study is $\mu$ the mean height of people in all the world. It doesnt make much sense saying that the probability of this mean height being between 160cm and 170cm is 95% because either this heigh is a number between 160-170cm or it is not. Even if we cannot calculate this mean height because we would require to survey all the people in the world, $\mu$ is still a fixed quantity, though an unknown one. Talking about probabilities for fixed numbers do not make much sense.

What we can do is take a sample of people and obtain a CI. A change of sample implies a change of CI. For this reason, if we obtain $100$ samples and compute $100$ confidence intervals at a $95\%$ level (one CI per sample), roughly speaking we can say that more or less $95$ confidence intervals would cover the unknown parameter $\mu$ and $5$ would not cover it. We do not know the value of $\mu$, so we do not know which are the CIs covering it. The only thing that we can say is that the probability that a confidence interval covers $\mu$ is $95\%$

Answer 2

The value of the population parameter is unknown and unknowable, and remains so after sampling. It is a fixed value (within a fixed coefficients framework), so that value either is or is not within any given interval. Aside from taking the value of an unbiased estimator to be a best guess for the value of the parameter, the information used to estimate a confidence interval all relates to the estimator, rather than to the parameter itself. Thus, a 95% confidence interval can describe the distribution of the estimator in repeated sampling but not the "distribution" of the (fixed) parameter. Repeated sampling will produce confidence intervals that will contain this range 95% of the time. That is the limit of the technology. Metrologists in the physical sciences and legal forensics resolve the problem (and others) by dealing not in standard errors but in uncertainty. (See the Guide to the Expression of Uncertainty in Measurement.) Uncertainty includes sampling variance as well as all other factors that materially contribute to doubt about the value of the measurand (the quantity sought). Again, unbiased estimator represents our best guess of the parameter's value, but uncertainty defines a range of plausible values for the parameter--not a distribution for the parameter, but a range of values for the parameter which are all consistent with our limited knowledge.

Answer 3

Confidence intervals answer the question: "What is the range of plausible findings under the triparte assumption that the population is Normally distributed, the mean is the observed value and the standard deviation is also at the observed value". So it's really saying "if you knew for certain that the data was an excellent representation of sampling of/from the population, what would be the range of means under a random sampling of similar size. So you first sentence is accurate but your second sentence:

Rather, if your sample was drawn an infinite number of times, 95% of the intervals would contain the population parameter (while 5% would not).

.... is not. What is being said is that IF the population parameter were mu, then the sampling distribution around mu have 95% coverage of mu-hats by the CI.

It's really that same duality as "type I errors" and "type II errors". They are calculated in two separate universes. Type II errors are those that occur under a hypothetical of a theoretical non-null situation (H[A]), whereas type I errors are those that occur in a theoretical null (H[0]) situation.

Until I figured this out I was bothered by the usual diagram of a Normal curve centered around the null hypothesis for "type I errors and a shifted Normal curve centered around the alternate hypothesis for type II errors. Turns out that the curve around the alternate hypothesis should NOT be a Normal curve, but rather should be a non-central t-distribution, which is NOT symmetric. (For any decent size problem the difference is pretty much undetectable, and you really can only see the difference when your sample sizes are less than 10.)