# Confidence interval in the linear model

Assume a Gaussian linear model $y=X\beta+\epsilon$ where $\epsilon \sim N(0,\sigma^2I_N)$. In Hastie's Elements of Statistical Inference, it is stated that a $1-2\alpha$ confidence interval for $\beta_j$ is $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma},\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma})$$ where $v_j=(X^TX)^{-1}_{jj}$ and $z^{(1-\alpha)}$ is the $1-\alpha$ percentile of standard normal distribution.

I'm wondering where $\hat{\sigma}$ comes from. I know that $\hat \beta \sim N(\beta,\sigma^2(X^TX)^{-1})$ hence $\hat{\beta}_j\sim N(\beta_j,\sigma^2v_j)$, thus the confidence interval should be $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\sigma,\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\sigma)$$ I don't get why there are some $\hat{\sigma}$ instead. Of course, $\sigma$ is unknown so the confidence interval would be useless... Is it standard practice to just replace $\sigma$ with $\hat{\sigma}$ ?

• Obviously you need some estimate of $\sigma$. Thus, is your question about which estimator $\hat\sigma$ to use or is it about why the confidence interval takes this particular form? – whuber Jul 4 '17 at 15:46
• To be perfectly correct, the confidence interval with variance estimator $\hat{\sigma}$ should use student-t $t^{1-\alpha}(n - p)$ percentile (if you use standard variance estimator based on $RSS$) – Łukasz Grad Jul 4 '17 at 16:05
• @whuber my question is about the form of the confidence interval: what's the mathematical validity of replacing $\sigma$ by something else ? – Gabriel Romon Jul 4 '17 at 16:15
• "Of course, $σ$ is unknown so the confidence interval would be useless" — There's your answer. – Kodiologist Jul 4 '17 at 18:36
• @Kodiologist so you just replace it by $\hat \sigma$ out of thin air ? There has to be a mathematical proof that you can do that – Gabriel Romon Jul 4 '17 at 18:37

$$\frac{X -\mu}{\sigma}$$ which would have a $N(0,1)$ distribution. Then you form your confidence interval (C.I.) by "pivoting" around two quantiles to ensure the correct coverage probability. Assume the quantiles are $97.5$ and $2.5$ for a 95% C.I. Then you get
$$\Pr(z_{2.5} <\frac{X -\mu}{\sigma}< z_{97.5})=95\%$$
and by ignoring the probability statement and just focusing on the inequalities you can solve for the $X$ which gives you the typical form of the C.I. Now because this example the O.P. asked about is also a normal distribution you can replace $X$ with $\hat{\beta}$ and if you $(X^TX)^{-1}$ is diagonal with entries $v_j$. Then you can do a similar inversion to get the stated C.I. However, as stated in the comments, if you estimated $\sigma^2$ with the usual estimator then the ratio above with $\hat{\sigma}$ then the distribution of the statistic is no longer normal but $t$ distributed and you should use the $t$ with the correct degrees of freedom. The degrees of freedom should be $n-p+1$ where $n$ is the number of observations, $p$ is the number of covariates you have and the $+1$ is for the intercept term-assuming you have one.