How to understand taking a simple sample to compute the Confidence Interval, using CLT?

Question

Suppose there is a population distribution having mean M and standard deviation SD. We want to estimate the Confidence Interval by using a single sample using CLT. We take a sample and compute it's mean(say SM) and standard deviation(say SSD). The distribution of Sample Means is normally distributed with a standard deviation close to SD/sqrt(n) where n is the sample size, and mean(say MSM) close to M.

We can understand this process in two ways:

1. That what we are trying to compute is the range MSM +/- 2SD/sqrt(n). This is what our Confidence Interval is.

   Now:

   • We approximate(replace) MSM with SM.

   • SD and SSD are quite close, so SD/sqrt(n) can be written as SSD/sqrt(n).

   Now, we can compute the Confidence Interval(MSM +/- 2SD/sqrt(n)), using the sample mean and sample standard deviation.

   But the problem with this understanding is that as a result of our process using CLT and a single sample, we can end up taking a sample having an exact mean of MSM + 2SD/sqrt(n) or MSM - 2SD/sqrt(n), with a 95% probability. In such cases, the Confidence Interval we will be computing are: MSM to MSM + 4SD/sqrt(n) or MSM - 4SD/sqrt(n) to MSM, which will be 50% off the confidence interval we were meant to calculate. So, how one solves this contradiction? The second way might have the answer.

2. That we assume MSM is actually quite close or same as M (Population Mean). Then we say that we have a 95% chance of taking out a single sample which has the mean within range MSM +/- 2SD/sqrt(n), in which case it will be 100% certain that our MSM(or Population Mean) is within 2SD/sqrt(n) of the SM (Sample Mean). In other words, there is a 95% chance of our Population Mean lying within 2SD/sqrt(n) of the SM. So, that is how it comes out to be our Confidence Interval.

But if the second approach is indeed correct, that would mean forgoing the idea that the distribution of Sample Means provides a Confidence Interval of the Population Mean(M) around it's mean(MSM). And will that be correct?

Thanks

Answer 1

So let's start by breaking down the formula for confidence intervals (CIs), Let's consider this in a little more generality then in your question. Let's say we want a confidence interval for an estimator of some parameter of the population, $\hat\theta$ (this could be the sample mean, or a regression coefficient, etc.). The population (true) parameter will be $\theta_0$. The CI is given by

$$[\hat\theta - c/\sqrt{n}, \hat\theta + c/\sqrt{n}]$$

$c$ is a critical value that we need to choose so that the CI will include $\theta_0$ with probability $1-\alpha$. This is our significance level. Setting $\alpha = 0.05$ is where you get that 95% figure. So what we want is that,

$$\Pr[\theta_0 \in [\hat\theta - c/\sqrt{n}, \hat\theta + c/\sqrt{n}]] = 1-\alpha$$ We can rewrite this probability as, $$\Pr[\sqrt{n}|\hat\theta - \theta_0|\leq c]$$

This is easy to see just subtract out $\hat\theta$ from the upper and lower bounds and multiply by $\sqrt{n}$.

Now I am not sure how much statistics that you have but $\sqrt{n}|\hat\theta-\theta_0|$ is a very common expression. In many estimators, this will have an asymptotic distribution that can be proven to be normal using the CLT. Given this normal asymptotic distribution, we can use what we know about normal distributions to pick a $c$ in which we have $1-\alpha$ coverage.

To see this in action let's go back to your example of normal means. Let's let ${X_1, \dots, X_n}\sim N(\mu, \sigma^2)$ and iid. Let us assume both the mean and $\sigma^2$ are known for simplicity. Then $\hat\theta=\bar X_n$ or the sample mean. Then we have,

$$\Pr[\frac{|\bar X_n - \mu|}{\sigma \sqrt{n}}\leq c/\sigma] = \Pr[|Z|\leq \frac{c}{\sigma}]$$

To see that notice that all we did was divide by $n$ and $\sigma$ on both sides (noticing both are positive). Now we immediately know that $\frac{|\bar X_n - \mu|}{\sigma \sqrt{n}}$ is a z-score and is distributed normal and so we replace it with $Z\sim N(0,1)$. In other cases we will use the fact $\sqrt{n}{(\hat\theta-\theta)}\to^d N(0,V)$ to do the same thing.

Now that we are this far all we need to do is invert the test and determine what $c$ should be for our desired confidence value. That is we want to find $z_\alpha$ such that,

$$\Pr[Z\leq z_\alpha]=\alpha$$

To do this we need to look at the inverse of the normal pdf which is what you do when you look through a Z-table. Once we find this value we can set $c/\sigma=\sigma z_{1-\alpha/2}$ (two-sided stat). To get the confidence set,

$$[\bar X_n - \frac{\sigma}{\sqrt{n}}z_{1-\alpha/2}, \bar X_n + \frac{\sigma}{\sqrt{n}}z_{1-\alpha/2}]$$

which will have $1-\alpha$ coverage.

So to wrap up this is where the normal approximation is coming from and how the CLT is used. You will notice we never replaced the population mean (parameter) with the sample mean (parameter). Rather we just found a region around the sample parameter for which we can be sure (to a $1-\alpha$ level of confidence) that the population parameter lives in using our normal approximation.

Answer 2

"M and SM are quite close and so, M can be replaced with SM." No, M is nowhere replaced by SM, and you don't need to assume they are close. The CLT makes a statement about the distribution, i.e., the variation of the SM, its distance from M. It doesn't require anywhere that this distance is small (although it is expected to become small for large n as a consequence of the theorem).

"Let us suppose that the normal distribution of sample means, has an exact mean of MSM." The mean of the distribution of the sample means is always M, even without CLT, normality, and asymptotics (as long as M exists).

"...what we are trying to compute is the range MSM +/- 2SD/sqrt(n)." No, we're fine with SM +/- 2SD/sqrt(n), no need to involve MSM.

"So, how one solves this contradiction?" By not introducing MSM in the first place.

"2. That we assume MSM is actually quite close or same as M (Population Mean)." These are the same anyway, see above.