Given a multiple regression with the usual assumptions satisfied, with $X \in R^{n \times p}$

$$ y = X \beta + e $$

I know that the estimated variance is given by $\sigma^2 (X^TX)^{-1}$. But what I want to know is the estimated variance for the $k^{th}$ coefficient out of this covariance matrix.

I've seen from the resource A (cited below) that this value is the $k^{th}$ diagonal term of the $\sigma^2 (X^TX)^{-1}$. But more interestingly (AND THIS IS WHAT I WANT TO PROVE), this value is:

$$ \dfrac{\sigma^2}{(1-R^2_k) \sum_{i=1}^n (x_{ik} - \bar{x_k})^2 } $$

Here, $x_k$ is the $k^{th}$ column of $X$. $R^2_k$ is the $R^2$ from regressing $x_k$ on $X_{(k)}$(=$X$ after taking $k^{th}$ column out).

Here is my (failed) attempt at this derivation:

Using resource B (cited below), the $k^{th}$ diagonal value of $\sigma^2 (X^TX)^{-1}$ is:

$$\sigma^2 [x_k^Tx_k - x_k^T X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k ] ^ {-1}$$

Since $X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k$ can be seen as projection of $x_k$ onto column space of $X_{(k)}$, we can say:

$$ X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k = \hat{x_k} $$, which is the regression prediction after regressing $x_k$ on $X_{(k)}$.

So our diagonal value simplifies to:

$$ \sigma^2 [x_k^Tx_k - x_k^T X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k ] ^ {-1}= \sigma^2 [ x_k^Tx_k - x_k^T \hat{x_k} ] ^ {-1}$$

Now, since (I will omit subscript $i$ so $x_k = x_{ik}$)

$$ R^2_k = 1 - \dfrac{ \sum (x_k - \hat{x_k} )^2 }{ \sum (x_k - \bar{x_k} )^2 } $$

$$ \dfrac{\sigma^2}{(1-R^2_k) \sum (x_{k} - \bar{x_k})^2 } $$

will simplify to:

$$ \dfrac{\sigma^2}{ \sum (x_{k} - \hat{x_k})^2 } $$

So in summary, we want to show that:

$$ \sigma^2 [ x_k^Tx_k - x_k^T \hat{x_k} ] ^ {-1} = \dfrac{\sigma^2}{ \sum (x_{k} - \hat{x_k})^2 }$$

But then,

$$ \sigma^2 [ x_k^Tx_k - x_k^T \hat{x_k} ] ^ {-1} = \dfrac{\sigma^2}{\sum (x_k)^2 - \sum x_k \hat{x_k} } $$

where as,

$$ \dfrac{\sigma^2}{ \sum (x_{k} - \hat{x_k})^2 } = \dfrac{\sigma^2}{\sum (x_k)^2 - 2\sum x_k \hat{x_k} + \sum \hat{x_k}^2 } $$

Could someone please help me find where I am making a mistake, and if this is not the right approach, help me derive it correctly?

Resource A: http://people.stern.nyu.edu/wgreene/MathStat/GreeneChapter4.pdf (page 40)

Resource B: https://math.stackexchange.com/questions/1562745/about-the-diagonal-entries-of-an-inverse-matrix


1 Answer 1


I will answer my own question:

It comes down to proving $$ \sum x_k \hat{x_k} = \sum \hat{x_k} \hat{x_k} $$

We can see this because $ x_k = \hat{x_k} + e_k $, where $e_k$ is the residual.

$$ \sum x_k \hat{x_k} = \sum (\hat{x_k} + e_k) \hat{x_k} $$

$$ = \sum \hat{x_k} \hat{x_k} + \sum e_k \hat{x_k}$$

But then $ \sum e_k \hat{x_k} = 0 $ since fitted values and residuals are orthogonal.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.