5
$\begingroup$

Assume a Gaussian linear model $y=X\beta+\epsilon$ where $\epsilon \sim N(0,\sigma^2I_N)$. In Hastie's Elements of Statistical Inference, it is stated that a $1-2\alpha$ confidence interval for $\beta_j$ is $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma},\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\hat{\sigma})$$ where $v_j=(X^TX)^{-1}_{jj}$ and $z^{(1-\alpha)}$ is the $1-\alpha$ percentile of standard normal distribution.

I'm wondering where $\hat{\sigma}$ comes from. I know that $\hat \beta \sim N(\beta,\sigma^2(X^TX)^{-1})$ hence $\hat{\beta}_j\sim N(\beta_j,\sigma^2v_j)$, thus the confidence interval should be $$(\hat{\beta}_j-z^{(1-\alpha)}\sqrt{v_j}\sigma,\hat{\beta}_j+z^{(1-\alpha)}\sqrt{v_j}\sigma)$$ I don't get why there are some $\hat{\sigma}$ instead. Of course, $\sigma$ is unknown so the confidence interval would be useless... Is it standard practice to just replace $\sigma$ with $\hat{\sigma}$ ?

$\endgroup$
4
  • 1
    $\begingroup$ Obviously you need some estimate of $\sigma$. Thus, is your question about which estimator $\hat\sigma$ to use or is it about why the confidence interval takes this particular form? $\endgroup$
    – whuber
    Commented Jul 4, 2017 at 15:46
  • 1
    $\begingroup$ To be perfectly correct, the confidence interval with variance estimator $\hat{\sigma}$ should use student-t $t^{1-\alpha}(n - p)$ percentile (if you use standard variance estimator based on $RSS$) $\endgroup$ Commented Jul 4, 2017 at 16:05
  • $\begingroup$ @whuber my question is about the form of the confidence interval: what's the mathematical validity of replacing $\sigma$ by something else ? $\endgroup$ Commented Jul 4, 2017 at 16:15
  • $\begingroup$ "Of course, $σ$ is unknown so the confidence interval would be useless" — There's your answer. $\endgroup$ Commented Jul 4, 2017 at 18:36

3 Answers 3

3
$\begingroup$

The confidence interval is an interval estimator for a parameter, so it must be a statistic ---i.e., it must be something we can compute entirely from the data. Since $\sigma$ is unknown we therefore need to replace it with some estimator $\hat{\sigma}$, and usually this is done using the (internal or external) RMSE in the regression model.

Actually, because we replace the true regression standard error $\sigma$ with the estimator $\hat{\sigma}$, arguably you should be using a different confidence interval based on the studentised pivotal quantity:

$$\frac{\hat{\beta}_j-\beta_j}{\sqrt{v_j} \cdot\hat{\sigma}} \sim \text{St}(\text{df}=df_\text{Res}),$$

where the notation $\text{St}$ to refer to the Student's T distribution and $df_\text{Res}$ is the residual degrees-of-freedom in the regression model. (This pivotal quantity is exact if $\hat{\sigma}$ is the externally studentised RMSE and inexact if $\hat{\sigma}$ is the internally studentised RMSE.) If you use this latter pivotal quantity (which is arguably more accurate for small to moderate residual degrees-of-freedom) then you get the confidence interval:

$$\text{CI}_{\beta_j}(1-\alpha) = \Bigg[ \hat{\beta}_j \pm t_{df_\text{Res}, \alpha/2} \cdot \sqrt{v}_j \cdot \hat{\sigma} \Bigg].$$

This is the usual confidence interval for the unknown coefficients in a regression; it is only equal to the one shown here in Hastie (i.e., the normal form) as $df_\text{Res} \rightarrow \infty$ (i.e., when the Student's T distribution converges to the normal distribution).

$\endgroup$
2
$\begingroup$

The confidence interval mentioned in your post with $\hat\sigma$ is only approximate in the sense that it is based on the following asymptotic distribution as the sample size $N\to \infty$:

$$\frac{\hat\beta_j-\beta_j}{\hat\sigma\sqrt{v_j}} \stackrel{d}\longrightarrow N(0,1) \tag{1}$$

Here $\hat\sigma$ can be any consistent estimator of $\sigma$.

To see why $(1)$ is true, start from the fact that as $N\to \infty$,

$$\frac{\hat\beta_j-\beta_j}{\sigma\sqrt{v_j}} \stackrel{d}\longrightarrow N(0,1) \tag{2}$$

Note that

$$\frac{\hat\beta_j-\beta_j}{\hat\sigma \sqrt{v_j}}=\frac{\sigma}{\hat\sigma}\cdot\frac{\hat\beta_j-\beta_j}{\sigma \sqrt{v_j}} $$

Since $\frac{\sigma}{\hat\sigma} \stackrel{P}\longrightarrow 1$ as $N\to \infty$, by Slutsky's theorem, $(2)$ implies $(1)$.

So both $\frac{\hat\beta_j-\beta_j}{\sigma\sqrt{v_j}}$ and $\frac{\hat\beta_j-\beta_j}{\hat\sigma\sqrt{v_j}}$ have identical limiting distributions as $N\to \infty$, which is a justification for replacing $\sigma$ by $\hat\sigma$ in the confidence interval. This process is sometimes called Studentization. The resulting confidence interval is a Wald-type confidence interval.

$\endgroup$
1
$\begingroup$

Based on the question the OP asked and the comments to clarify exactly what the OP is looking for I can sketch out the proof. Anytime you form a confidence interval you want base it off of a statistic that has a distribution that is not dependent on the statistic. For things with normal distributions this is usually something like

$$\frac{X -\mu}{\sigma}$$ which would have a $N(0,1)$ distribution. Then you form your confidence interval (C.I.) by "pivoting" around two quantiles to ensure the correct coverage probability. Assume the quantiles are $97.5$ and $2.5$ for a 95% C.I. Then you get

$$\Pr(z_{2.5} <\frac{X -\mu}{\sigma}< z_{97.5})=95\% $$

and by ignoring the probability statement and just focusing on the inequalities you can solve for the $X$ which gives you the typical form of the C.I. Now because this example the O.P. asked about is also a normal distribution you can replace $X$ with $\hat{\beta}$ and if you $(X^TX)^{-1}$ is diagonal with entries $v_j$. Then you can do a similar inversion to get the stated C.I. However, as stated in the comments, if you estimated $\sigma^2$ with the usual estimator then the ratio above with $\hat{\sigma}$ then the distribution of the statistic is no longer normal but $t$ distributed and you should use the $t$ with the correct degrees of freedom. The degrees of freedom should be $n-p+1$ where $n$ is the number of observations, $p$ is the number of covariates you have and the $+1$ is for the intercept term-assuming you have one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.