Is 95% specific to the confidence interval in any way?

Question

I am aware of the misconception that a "95% confidence interval means there is a 95% chance that the true parameter falls in this range," and that the correct interpretation is that if you build, say, 100, of these confidence intervals from random sampling, then 95 of the confidence intervals should include the true parameter.

In https://www.econometrics-with-r.org/5-2-cifrc.html, I see the following:

Is this wording incorrect? It seems to be saying that the true value has a 95% chance of being in that specific confidence interval.

My second question is, say you have one of these 95 confidence intervals. Aside from using 95% to get the 1.96 Z-score, how else is the 95% manifested in this confidence interval?

Answer 1

Is this wording incorrect? It seems to be saying that the true value has a 95% chance of being in that specific confidence interval.

You have to keep in mind that, in frequentist statistics, the parameter you are estimating (in your case $\beta_i$, the true value of the coefficient) is not considered as a random variable, but as a fixed real number. That means it is not correct to say something like "$\beta_i$ is in the interval $[a,b]$ with $95\%$ probability", because $\beta_i$ is not a random variable and therefore does not have a probability distribution. The probability of $\beta_i$ being in the interval is either $100\%$ (if the fixed value $\beta_i\in[a,b]$) or $0\%$ (if the fixed value $\beta_i\notin[a,b]$)

That is why "95% confidence interval means there is a 95% chance that the true parameter falls in this range" is a misconception.

On the other hand, the limits of the confidence interval themselves are random variables, since they are calculated from the sample data. That means it is correct to say "in 95% of all possible samples, $\beta_i$ is in the 95% confidence interval". It does not mean that $\beta_i$ has $95\%$ chance of being inside a particular interval, it means that the confidence interval, which is different for each sample, has $95\%$ probability of falling around $\beta_i$.

Notice that the confidence interval will contain $\beta_i$ with 95% probability before the data is sampled. After it is sampled, the confidence intervals edges will be just two fixed numbers, not random variables anymore and the same rationale from the first paragraph applies. I think the following image offers a nice visualization to this idea:

Therefore, the wording used there is actually correct.

Aside from using 95% to get the 1.96 Z-score, how else is the 95% manifested in this confidence interval?

The 1.96 Z-score is the only place where the 95% shows up. If you change it for the Z-score corresponding to, say, 85%, you would have the formula 85% confidence interval.

Answer 2

Perhaps if you rephrase to:

"Imagine you repeat your sampling under the exact same conditions indefinitely. For each draw you calculate a parameter estimate and its standard error in order to calculate a 95% confidence interval [formula in your figure]. Then this 95% confidence interval will capture the true population parameter in 95% of the time if all assumptions are met and the null hypothesis is true."

Would that make more sense?

As for you second question, consider the standard normal distribution below. The total area under the curve equals to 1. If you consider the significance level to be 5% and split this up between each tail (red areas), then you are left with 95% in the middle. If the null hypothesis is true then this is the area in which you would not reject the null hypothesis as any Z-score that falls in that area is plausible under the null hypothesis. Only if your Z-score falls into the red areas, you reject the null hypothesis, since your sample provides significant evidence against the null hypothesis, or in other words you likely made a discovery - hooray :D

Now by multiplying the critical Z-score of +/-1.96 (in case of 95% confidence) with the standard error of the sample you are translating this 95% interval back onto the original measurement scale. So each confidence interval corresponds to a hypothesis test on your measurement scale as suggested in the last sentence on your image.

Answer 3

95% conf.int. means there is only a 5% chance that actual empirical value falls out of this interval. In other words, 5% chance of false positive if (and when) you treat that range as ground truth.

Answer 4

My second question is, say you have one of these 95 confidence intervals. Aside from using 95% to get the 1.96 Z-score, how else is the 95% manifested in this confidence interval?

The 95% is manifested in the following way

Probability statement: "95% confidence interval means there is a 95% chance that the true parameter falls in this range"

There are misconceptions about this statement, but the statement itselve is not a misconception.

Whether or not the probability statement is true depends on how you condition the probability. It depends on the interpretation of what is meant by probability/chance.

You can express this probability 'the interval containing the true parameter' conditional on the observation but also conditional on the true parameter.

So yes, the probability statement is wrong if the probability is wrongly interpreted.

But no, the probability statement is not wrong if the probability is correctly interpreted.

Example

from https://stats.stackexchange.com/a/481937/ and https://stats.stackexchange.com/a/444020/

Say we measure $X$ in order to determine/estimate $\theta$

$$X \sim N(\theta,1) \quad \text{where} \quad \theta \sim N(0,\tau^2)$$

Here $\theta$ follows a distribution as well. (You can imagine for instance that $\theta$ is some measure of intelligence which differs from person to person where $N(0,\tau^2)$ is the distribution of $\theta$ among all persons. And $X$ is the result from some intelligence test).

Below is a simulation of 20k cases.

In the image we draw lines for the borders of a 95% confidence interval (red) and a 95% credible interval (green) as a function of the observation $X$. (For more details about the computation of those borders see the reference)

We can consider 'the probability of an interval containing the true parameter' as conditional on the observation $X$ or conditional on the true parameter $\theta$. We have plotted the two different interpretations for both type of intervals. Only in the right one (conditional on $X$), the probability statement about the confidence interval is false.

.

Practical situation that explains the relevance of this view: Say in the above example, which could be about an intelligence test, we have done the test in order to select people with high intelligence, and we have a sub sample of people that tested with a specific range of test observation $X$ (e.g. we selected candidates with a high intelligence for some job). Now we may wonder for how many of these people their 95% confidence interval contains the true parameter.... well, it won't be 95% because the confidence interval does not contain the parameter in 95% of the cases when we condition on a particular observation (or range of observations).

Misconceptions about the misconception

It is often mentioned that the probability statement is not true because after the observation the statement is either 100% true or 0% true. The parameter is either in or outside the interval and it can not be both. But we do not know which of the two cases it is, and we express a probability for our data-based certainty/uncertainty about it (based on some assumptions).

(Note that this argument about 'the probability statement being false' would work for any type of interval, but somehow because the confidence interval relates to a frequentist interpretation of probability the probability statement is disallowed.)

It is perfectly fine to speak about the probability that 'the parameter is within some interval' (Or if you feel uncomfortable with this expression then you can turn it around and say, the probability that 'the interval is around the parameter') Even if the parameter would be a constant, the interval is not a constant (but a random variable), thus it is fine to make probability statements about relations between the two. (and if this argument is still not comfortable then one could also adopt a propensity probability interpretation instead of a frequentist interpretation of probability, a frequentist interpretation is not necessary for confidence intervals)

Example: from an urn with 10 red and 10 blue balls you 'randomly' pick a ball without looking at it. Then you could say that you have 50% probability to have picked red and 50% to have picked blue. Even though in the (unknown) reality it is either 100% blue or 100% red and not really a random pick but a deterministic process.

Pedantic notes

The interval that contains the true value $\beta_i$ in 95% of all samples is given by the expression...

Is this wording correct?

• It is not the interval. There will be multiple intervals with the same property.
• More specifically it is an interval that contains the true value $\beta_i$ in 95% of all samples independent from the true value of $\beta_i$