# Confidence interval in the linear model

Assume a Gaussian linear model $y=X\beta+\epsilon$ where $\epsilon \sim N(0,\sigma^2I_N)$. In Hastie's Elements of Statistical Inference, it is stated that a $1-2\alpha$ confidence interval for $\beta_j$ is $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma},\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma})$$ where $v_j=(X^TX)^{-1}_{jj}$ and $z^{(1-\alpha)}$ is the $1-\alpha$ percentile of standard normal distribution.

I'm wondering where $\hat{\sigma}$ comes from. I know that $\hat \beta \sim N(\beta,\sigma^2(X^TX)^{-1})$ hence $\hat{\beta}_j\sim N(\beta_j,\sigma^2v_j)$, thus the confidence interval should be $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\sigma,\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\sigma)$$ I don't get why there are some $\hat{\sigma}$ instead. Of course, $\sigma$ is unknown so the confidence interval would be useless... Is it standard practice to just replace $\sigma$ with $\hat{\sigma}$ ?

• Obviously you need some estimate of $\sigma$. Thus, is your question about which estimator $\hat\sigma$ to use or is it about why the confidence interval takes this particular form?
– whuber
Commented Jul 4, 2017 at 15:46
• To be perfectly correct, the confidence interval with variance estimator $\hat{\sigma}$ should use student-t $t^{1-\alpha}(n - p)$ percentile (if you use standard variance estimator based on $RSS$) Commented Jul 4, 2017 at 16:05
• @whuber my question is about the form of the confidence interval: what's the mathematical validity of replacing $\sigma$ by something else ? Commented Jul 4, 2017 at 16:15
• "Of course, $σ$ is unknown so the confidence interval would be useless" — There's your answer. Commented Jul 4, 2017 at 18:36

## 3 Answers

The confidence interval is an interval estimator for a parameter, so it must be a statistic ---i.e., it must be something we can compute entirely from the data. Since $$\sigma$$ is unknown we therefore need to replace it with some estimator $$\hat{\sigma}$$, and usually this is done using the (internal or external) RMSE in the regression model.

Actually, because we replace the true regression standard error $$\sigma$$ with the estimator $$\hat{\sigma}$$, arguably you should be using a different confidence interval based on the studentised pivotal quantity:

$$\frac{\hat{\beta}_j-\beta_j}{\sqrt{v_j} \cdot\hat{\sigma}} \sim \text{St}(\text{df}=df_\text{Res}),$$

where the notation $$\text{St}$$ to refer to the Student's T distribution and $$df_\text{Res}$$ is the residual degrees-of-freedom in the regression model. (This pivotal quantity is exact if $$\hat{\sigma}$$ is the externally studentised RMSE and inexact if $$\hat{\sigma}$$ is the internally studentised RMSE.) If you use this latter pivotal quantity (which is arguably more accurate for small to moderate residual degrees-of-freedom) then you get the confidence interval:

$$\text{CI}_{\beta_j}(1-\alpha) = \Bigg[ \hat{\beta}_j \pm t_{df_\text{Res}, \alpha/2} \cdot \sqrt{v}_j \cdot \hat{\sigma} \Bigg].$$

This is the usual confidence interval for the unknown coefficients in a regression; it is only equal to the one shown here in Hastie (i.e., the normal form) as $$df_\text{Res} \rightarrow \infty$$ (i.e., when the Student's T distribution converges to the normal distribution).

The confidence interval mentioned in your post with $$\hat\sigma$$ is only approximate in the sense that it is based on the following asymptotic distribution as the sample size $$N\to \infty$$:

$$\frac{\hat\beta_j-\beta_j}{\hat\sigma\sqrt{v_j}} \stackrel{d}\longrightarrow N(0,1) \tag{1}$$

Here $$\hat\sigma$$ can be any consistent estimator of $$\sigma$$.

To see why $$(1)$$ is true, start from the fact that as $$N\to \infty$$,

$$\frac{\hat\beta_j-\beta_j}{\sigma\sqrt{v_j}} \stackrel{d}\longrightarrow N(0,1) \tag{2}$$

Note that

$$\frac{\hat\beta_j-\beta_j}{\hat\sigma \sqrt{v_j}}=\frac{\sigma}{\hat\sigma}\cdot\frac{\hat\beta_j-\beta_j}{\sigma \sqrt{v_j}}$$

Since $$\frac{\sigma}{\hat\sigma} \stackrel{P}\longrightarrow 1$$ as $$N\to \infty$$, by Slutsky's theorem, $$(2)$$ implies $$(1)$$.

So both $$\frac{\hat\beta_j-\beta_j}{\sigma\sqrt{v_j}}$$ and $$\frac{\hat\beta_j-\beta_j}{\hat\sigma\sqrt{v_j}}$$ have identical limiting distributions as $$N\to \infty$$, which is a justification for replacing $$\sigma$$ by $$\hat\sigma$$ in the confidence interval. This process is sometimes called Studentization. The resulting confidence interval is a Wald-type confidence interval.

Based on the question the OP asked and the comments to clarify exactly what the OP is looking for I can sketch out the proof. Anytime you form a confidence interval you want base it off of a statistic that has a distribution that is not dependent on the statistic. For things with normal distributions this is usually something like

$$\frac{X -\mu}{\sigma}$$ which would have a $N(0,1)$ distribution. Then you form your confidence interval (C.I.) by "pivoting" around two quantiles to ensure the correct coverage probability. Assume the quantiles are $97.5$ and $2.5$ for a 95% C.I. Then you get

$$\Pr(z_{2.5} <\frac{X -\mu}{\sigma}< z_{97.5})=95\%$$

and by ignoring the probability statement and just focusing on the inequalities you can solve for the $X$ which gives you the typical form of the C.I. Now because this example the O.P. asked about is also a normal distribution you can replace $X$ with $\hat{\beta}$ and if you $(X^TX)^{-1}$ is diagonal with entries $v_j$. Then you can do a similar inversion to get the stated C.I. However, as stated in the comments, if you estimated $\sigma^2$ with the usual estimator then the ratio above with $\hat{\sigma}$ then the distribution of the statistic is no longer normal but $t$ distributed and you should use the $t$ with the correct degrees of freedom. The degrees of freedom should be $n-p+1$ where $n$ is the number of observations, $p$ is the number of covariates you have and the $+1$ is for the intercept term-assuming you have one.