Assume a Gaussian linear model $y=X\beta+\epsilon$ where $\epsilon \sim N(0,\sigma^2I_N)$. In Hastie's Elements of Statistical Inference, it is stated that a $1-2\alpha$ confidence interval for $\beta_j$ is $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma},\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma})$$ where $v_j=(X^TX)^{-1}_{jj}$ and $z^{(1-\alpha)}$ is the $1-\alpha$ percentile of standard normal distribution.

I'm wondering where $\hat{\sigma}$ comes from. I know that $\hat \beta \sim N(\beta,\sigma^2(X^TX)^{-1})$ hence $\hat{\beta}_j\sim N(\beta_j,\sigma^2v_j)$, thus the confidence interval should be $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\sigma,\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\sigma)$$ I don't get why there are some $\hat{\sigma}$ instead. Of course, $\sigma$ is unknown so the confidence interval would be useless... Is it standard practice to just replace $\sigma$ with $\hat{\sigma}$ ?

  • $\begingroup$ Obviously you need some estimate of $\sigma$. Thus, is your question about which estimator $\hat\sigma$ to use or is it about why the confidence interval takes this particular form? $\endgroup$ – whuber Jul 4 '17 at 15:46
  • $\begingroup$ To be perfectly correct, the confidence interval with variance estimator $\hat{\sigma}$ should use student-t $t^{1-\alpha}(n - p)$ percentile (if you use standard variance estimator based on $RSS$) $\endgroup$ – Łukasz Grad Jul 4 '17 at 16:05
  • $\begingroup$ @whuber my question is about the form of the confidence interval: what's the mathematical validity of replacing $\sigma$ by something else ? $\endgroup$ – Gabriel Romon Jul 4 '17 at 16:15
  • $\begingroup$ "Of course, $σ$ is unknown so the confidence interval would be useless" — There's your answer. $\endgroup$ – Kodiologist Jul 4 '17 at 18:36
  • $\begingroup$ @Kodiologist so you just replace it by $\hat \sigma$ out of thin air ? There has to be a mathematical proof that you can do that $\endgroup$ – Gabriel Romon Jul 4 '17 at 18:37

Based on the question the OP asked and the comments to clarify exactly what the OP is looking for I can sketch out the proof. Anytime you form a confidence interval you want base it off of a statistic that has a distribution that is not dependent on the statistic. For things with normal distributions this is usually something like

$$\frac{X -\mu}{\sigma}$$ which would have a $N(0,1)$ distribution. Then you form your confidence interval (C.I.) by "pivoting" around two quantiles to ensure the correct coverage probability. Assume the quantiles are $97.5$ and $2.5$ for a 95% C.I. Then you get

$$\Pr(z_{2.5} <\frac{X -\mu}{\sigma}< z_{97.5})=95\% $$

and by ignoring the probability statement and just focusing on the inequalities you can solve for the $X$ which gives you the typical form of the C.I. Now because this example the O.P. asked about is also a normal distribution you can replace $X$ with $\hat{\beta}$ and if you $(X^TX)^{-1}$ is diagonal with entries $v_j$. Then you can do a similar inversion to get the stated C.I. However, as stated in the comments, if you estimated $\sigma^2$ with the usual estimator then the ratio above with $\hat{\sigma}$ then the distribution of the statistic is no longer normal but $t$ distributed and you should use the $t$ with the correct degrees of freedom. The degrees of freedom should be $n-p+1$ where $n$ is the number of observations, $p$ is the number of covariates you have and the $+1$ is for the intercept term-assuming you have one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.