Two questions about confidence intervals

Question

I am learning about confidence intervals, but don't think I understand them very welll.

Assume $$(\mu - \hat{\mu}) \sqrt{\frac{n}{\sigma(\mu)}}$$ is asymptotically standard normal. So I guess we can say that a 95 % CI is $\hat{\mu} \pm 1.96 \sqrt{\sigma(\mu)/n}$.

I am a bit confused with respect to this, since $\sigma$ is a function of $\mu$.

1. Assume $\mu$ is unknown. How can we form a confidence interval given that $\sigma$ depends on $\mu$? I know we can just plug in $\hat{\mu}$ and pray it's close, but what if we don't want to do that? What are the alternatives, if any? What do people do in practice?

2. Assume $\mu$ is known. What is the interpretation of $\hat{\mu} \pm 1.96 \sqrt{\sigma(\mu)/n}$? I mean, if I know what $\mu$ is, does it still make sense to talk about confidence intervals around $\mu$? Isn't a "100 %" confidence interval then $[\mu, \mu]$, if that makes sense?

Answer 1

The whole game is about learning something about $\mu$, so you're right in your first question: the starting point is that $\mu$ is unknown and you want to use data and an estimator to learn about it. Using a given dataset, you can form an estimator $\hat \mu$ to estimate the value of $\mu$.

Formally, we usually think about $\mu$ as being a parameter (for instance, the expectation of a random variable, or the coefficient in a linear regression), and the estimator $\hat \mu$ as being a random variable, which depends on the realisation of the data. For a given dataset, you will obtain a given estimate.

Let us assume (as you do) that $\hat \mu$ is normally distributed. Its expectation is equal to $\mu$ (which happens when the estimator is consistent). Expectation, in this case, means that if you were to observe not one dataset, but a large number of datasets, the average value of the $\hat \mu$ over these datasets would be equal to $\mu$.

The variance of $\hat \mu$ is a function of two quantities: $\sigma$ the asymptotic/underlying variance, and $n$ the number of observations. $\sigma$ essentially depends on the data generating process of the random variable (for instance, the variance of the underlying random variable). Variance of $\hat \mu$ means: how would my $\hat \mu$ vary if I computed it many times, on many datasets of size $n$? If $n$ was very large, all the $\hat \mu$ would be pretty close to each other (and pretty close to $\mu$).

Now that we have all this, let's answer your question. You're right that the formula of the variance (and CI) of $\hat \mu$ depends on $\sigma$ and we don't observe $\sigma$. What is usually done is to plug an estimator of $\sigma$ instead. In many cases, just like you can form an estimator of $\mu$, you can form an estimator of $\sigma$, that you call $\hat \sigma$, which can be computed as a function of the data.

For instance, the canonical problem is that you have a random variable $Y_i$ distributed in a $\mathcal N(\mu, \sigma^2)$. In this case, we can take $\hat \mu$ to be the average of the observed $Y_i$. An unbiased estimator of the variance $\sigma^2$ is then: $$ \hat \sigma = \frac{\sum_i Y_i^2}{n} - \hat \mu^2 $$ Note that $\hat \sigma$ depends on $n$ and on the observations of the dataset $\{Y_i\}$ and on the estimator $\hat \mu$ (which also depends on $\{Y_i\}$ and $n$), but not on $\mu$. $\hat \sigma$ is the quantity you will plug into your confidence interval.