Confidence Interval - Revisit [duplicate]

I've read in many articles about Confidence Interval as below

One such article link: https://www.statisticssolutions.com/misconceptions-about-confidence-intervals/

1. [FALSE] - There is a 95% chance that the true population mean falls within the confidence interval.

2. [TRUE] - 95% of the confidence intervals calculated from these random samples will contain the true population mean.

Aren't both the statements the same?

Thought Process We plan to take 100 sets of random samples. From the TRUE point 2, around 95 of those intervals (good set) will contain the True population mean while around 5 intervals (bad set) will not contain the mean.

For the first set, we got a confidence interval, say [c1_start, c1_end]. The chance of this confidence interval belonging to the good set is 95 out of 100, and if it's in this set, it'll contain the true population mean. Thus, there is 95% chance/probability that the confidence interval [c1_start, c1_end] will contain the true population mean which is the 1st statement.

How come then the first statement is considered FALSE? Which part of my thought process is incorrect?

Based on the excerpt from Introduction to Statistical Learning The first point is True, or did I understand it wrong A 95% confidence interval is defined as a range of values such that with 95% probability, the range will contain the true unknown value of the parameter

My other question is if Confidence Interval can only be explained if we do the experiments many-many times, what does a single confidence interval tells me?

2 Answers

I view this as a philosophical question with no uniformly satisfactory answer.

Consider a 95% CI for $$\mu$$ based on a random sample of size $$n$$ from $$\mathsf{Norm}(\mu, \sigma),$$ where $$\sigma$$ is known. Before data are observed, there agreement among frequentist statisticians that $$P\left(\frac{|\bar X - \mu|}{\sigma/\sqrt{n}} \le 1.96\right)= P\left(\bar X - 1.96\frac{\sigma}{\sqrt{n}} \le \mu \le \bar X + 1.96\frac{\sigma}{\sqrt{n}}\right) = 0.95.$$

However, after data are available, there is disagreement whether the above remains a true probability statement. One can argue that the word "confidence" (instead of "probability") came into use because it is sufficiently vague to avoid arguments. About all one can say for sure is that, over the long run, the procedure giving rise to the interval $$\bar X \pm 1.96\frac{\sigma}{\sqrt{n}}$$ will cover $$\mu$$ 95% of the time. References to what this interval means for the specific experiment at hand are bound to provoke unproductive arguments.

When frequentist statistical consultants tell clients that a given CI is "95% sure" to include the true value of value of $$\mu,$$ they can feel safe because the exact true value of $$\mu$$ is typically never known.

In a Bayesian context, a prior distribution established a probability framework. A Bayesian posterior probability (or 'credible') interval, based on the prior and likelihood function from data, can be taken as a true probability statement about the current data. If you believe the prior and the integrity of the likelihood, there can be no quibbling about the resulting probability statement about the posterior distribution.

A confidence interval for a parameter $$\theta$$ is a set of plausible true values of $$\theta$$, given the observed data. In your example, $$\theta$$ would be the unknown population mean.

In frequentist statistics probability represents the long-run proportion of events as the number of samples tends to infinity. In this case, the proportion of intervals that contain the unknown fixed true mean. Our confidence in any one interval covering the truth comes from this long-run performance, but each individual interval either covers or it does not. In terms of a willingness to bet you would be willing to bet $$\0.95$$ and expect $$\1$$ in return if it can be determined that an observed interval in question covers the truth. This is based on the long-run performance of the interval over many experiments.

It may also be useful to think in terms of a free throw percentage. Suppose you did not witness the last free throw of your favorite player. He of course either made it or missed it. Your confidence in his performance on the last free throw would be based on his free throw percentage.

It comes down to how one defines probability and what it means for something to be random. Consider a coin toss. What does it mean for a coin to be fair, to have a probability of heads equal to $$50\%$$? Is it fair because I have $$50\%$$ belief it will land on heads? Is it fair because half $$(50\%)$$ of the coin is heads and the other half $$(50\%)$$ is tails? Is it fair because in many repeated flips it lands on heads $$50\%$$ of the time? The last definition is the only one that is defensible.

To the frequentist it is the act of sampling that constitutes randomness - flipping a coin, shooting a basketball, etc. Probability is the long-run proportion of events - the proportion of heads, proportion of baskets, etc. If probability was applicable to a single event, what would it mean? If I flip a fair coin, catch it, and cover it on the back of my hand, what would it mean to say the coin on my hand is heads with $$50\%$$ probability? Is the coin in a state of superposition just because we can not see it (think Schrodinger's cat)? No. It is either heads or it is tails. All we can discuss is the performance of the coin over many flips and place our bets accordingly. Returning to the basketball example one might say, "The probability that the player makes his next shot is $$75\%$$." This really means, "The player makes $$75\%$$ of his shots and this is one such shot."

Bayesians interpret probability as the belief of the experimenter. They use this to make belief statements about the parameter itself, rather than long-run probabilities about the experiment. The problem here is that probability becomes unfalsifiable. Returning to the coin example, if we interpret probability as belief then the coin can have any probability of landing heads depending on who you ask. No one is wrong because no one can be told they are not believing the right way. We can apply probability statements to anything unknown, perhaps even the flipped coin that has already landed on my hand. None of this is defensible.

Probability is a proportion, so ask yourself, proportion of what?

The flip side of a confidence interval is the p-value. This tells us how often we would observe a result like that witnessed if a particular hypothesis is true. This sampling probability tells us how plausible the hypothesis is based on the observed data. Returning to the coin example, we will only get to flip it a finite number of times. We can test various hypotheses about the long-run performance of the coin based on our sample using p-values. Those hypotheses that are not significant at the two-sided $$\alpha$$ level form the $$100(1-\alpha)\%$$ confidence interval. It is a set in the parameter space determined by the data. This last point is often overlooked when working with a closed-form interval like a Wald interval because it is written in terms of the sample mean and standard error. I find confidence curves to be a great way to visualize frequentist inference.

• The free throw example doesn't make things much clearer for me. If a player has historically made 95% of their free throws, I'd say they have a 95% probability of making their next free throw. I'd also say they have a 95% probability of having made their last free throw which I didn't see. Is that second interpretation incorrect? It's not clear to me how the interpretation changes depending on if I'm watching the game live (the free throw hasn't happened yet) or if it turns out to be a recording (the free throw actually had already happened). Aug 10, 2021 at 16:18
• This is a great question. It comes down to how one defines probability and what it means for something to be random. I'll add more to my answer above. Aug 10, 2021 at 17:02
• The only defensible definition of probability that addresses both the live game and the recorded game is that probability is the long-run free throw percentage of the athlete. It is not applicable to the live shot and it is not applicable to the recorded shot. But it does give us confidence in whether he will make the shot, whether we're watching live or we're watching a recording. Aug 10, 2021 at 17:32
• One might say, "The probability that the player makes his next shot is $75\%$." This really means, "The player makes $75\%$ of his shots and this is one such shot." Aug 11, 2021 at 13:39