Confidence interval in the linear model

Question

Assume a Gaussian linear model $y=X\beta+\epsilon$ where $\epsilon \sim N(0,\sigma^2I_N)$. In Hastie's Elements of Statistical Inference, it is stated that a $1-2\alpha$ confidence interval for $\beta_j$ is $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma},\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma})$$ where $v_j=(X^TX)^{-1}_{jj}$ and $z^{(1-\alpha)}$ is the $1-\alpha$ percentile of standard normal distribution.

I'm wondering where $\hat{\sigma}$ comes from. I know that $\hat \beta \sim N(\beta,\sigma^2(X^TX)^{-1})$ hence $\hat{\beta}_j\sim N(\beta_j,\sigma^2v_j)$, thus the confidence interval should be $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\sigma,\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\sigma)$$ I don't get why there are some $\hat{\sigma}$ instead. Of course, $\sigma$ is unknown so the confidence interval would be useless... Is it standard practice to just replace $\sigma$ with $\hat{\sigma}$ ?

Answer 1

The confidence interval is an interval estimator for a parameter, so it must be a statistic ---i.e., it must be something we can compute entirely from the data. Since $\sigma$ is unknown we therefore need to replace it with some estimator $\hat{\sigma}$, and usually this is done using the (internal or external) RMSE in the regression model.

Actually, because we replace the true regression standard error $\sigma$ with the estimator $\hat{\sigma}$, arguably you should be using a different confidence interval based on the studentised pivotal quantity:

$$\frac{\hat{\beta}_j-\beta_j}{\sqrt{v_j} \cdot\hat{\sigma}} \sim \text{St}(\text{df}=df_\text{Res}),$$

where the notation $\text{St}$ to refer to the Student's T distribution and $df_\text{Res}$ is the residual degrees-of-freedom in the regression model. (This pivotal quantity is exact if $\hat{\sigma}$ is the externally studentised RMSE and inexact if $\hat{\sigma}$ is the internally studentised RMSE.) If you use this latter pivotal quantity (which is arguably more accurate for small to moderate residual degrees-of-freedom) then you get the confidence interval:

$$\text{CI}_{\beta_j}(1-\alpha) = \Bigg[ \hat{\beta}_j \pm t_{df_\text{Res}, \alpha/2} \cdot \sqrt{v}_j \cdot \hat{\sigma} \Bigg].$$

This is the usual confidence interval for the unknown coefficients in a regression; it is only equal to the one shown here in Hastie (i.e., the normal form) as $df_\text{Res} \rightarrow \infty$ (i.e., when the Student's T distribution converges to the normal distribution).

Answer 2

Based on the question the OP asked and the comments to clarify exactly what the OP is looking for I can sketch out the proof. Anytime you form a confidence interval you want base it off of a statistic that has a distribution that is not dependent on the statistic. For things with normal distributions this is usually something like

$$\frac{X -\mu}{\sigma}$$ which would have a $N(0,1)$ distribution. Then you form your confidence interval (C.I.) by "pivoting" around two quantiles to ensure the correct coverage probability. Assume the quantiles are $97.5$ and $2.5$ for a 95% C.I. Then you get

$$\Pr(z_{2.5} <\frac{X -\mu}{\sigma}< z_{97.5})=95\% $$

and by ignoring the probability statement and just focusing on the inequalities you can solve for the $X$ which gives you the typical form of the C.I. Now because this example the O.P. asked about is also a normal distribution you can replace $X$ with $\hat{\beta}$ and if you $(X^TX)^{-1}$ is diagonal with entries $v_j$. Then you can do a similar inversion to get the stated C.I. However, as stated in the comments, if you estimated $\sigma^2$ with the usual estimator then the ratio above with $\hat{\sigma}$ then the distribution of the statistic is no longer normal but $t$ distributed and you should use the $t$ with the correct degrees of freedom. The degrees of freedom should be $n-p+1$ where $n$ is the number of observations, $p$ is the number of covariates you have and the $+1$ is for the intercept term-assuming you have one.

Answer 3

The confidence interval mentioned in your post with $\hat\sigma$ is only approximate in the sense that it is based on the following asymptotic distribution as the sample size $N\to \infty$:

$$\frac{\hat\beta_j-\beta_j}{\hat\sigma\sqrt{v_j}} \stackrel{d}\longrightarrow N(0,1) \tag{1}$$

Here $\hat\sigma$ can be any consistent estimator of $\sigma$.

To see why $(1)$ is true, start from the fact that as $N\to \infty$,

$$\frac{\hat\beta_j-\beta_j}{\sigma\sqrt{v_j}} \stackrel{d}\longrightarrow N(0,1) \tag{2}$$

Note that

$$\frac{\hat\beta_j-\beta_j}{\hat\sigma \sqrt{v_j}}=\frac{\sigma}{\hat\sigma}\cdot\frac{\hat\beta_j-\beta_j}{\sigma \sqrt{v_j}} $$

Since $\frac{\sigma}{\hat\sigma} \stackrel{P}\longrightarrow 1$ as $N\to \infty$, by Slutsky's theorem, $(2)$ implies $(1)$.

So both $\frac{\hat\beta_j-\beta_j}{\sigma\sqrt{v_j}}$ and $\frac{\hat\beta_j-\beta_j}{\hat\sigma\sqrt{v_j}}$ have identical limiting distributions as $N\to \infty$, which is a justification for replacing $\sigma$ by $\hat\sigma$ in the confidence interval. This process is sometimes called Studentization. The resulting confidence interval is a Wald-type confidence interval.