# How to derive the kth coefficient standard error?

Given a multiple regression with the usual assumptions satisfied, with $$X \in R^{n \times p}$$

$$y = X \beta + e$$

I know that the estimated variance is given by $$\sigma^2 (X^TX)^{-1}$$. But what I want to know is the estimated variance for the $$k^{th}$$ coefficient out of this covariance matrix.

I've seen from the resource A (cited below) that this value is the $$k^{th}$$ diagonal term of the $$\sigma^2 (X^TX)^{-1}$$. But more interestingly (AND THIS IS WHAT I WANT TO PROVE), this value is:

$$\dfrac{\sigma^2}{(1-R^2_k) \sum_{i=1}^n (x_{ik} - \bar{x_k})^2 }$$

Here, $$x_k$$ is the $$k^{th}$$ column of $$X$$. $$R^2_k$$ is the $$R^2$$ from regressing $$x_k$$ on $$X_{(k)}$$(=$$X$$ after taking $$k^{th}$$ column out).

Here is my (failed) attempt at this derivation:

Using resource B (cited below), the $$k^{th}$$ diagonal value of $$\sigma^2 (X^TX)^{-1}$$ is:

$$\sigma^2 [x_k^Tx_k - x_k^T X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k ] ^ {-1}$$

Since $$X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k$$ can be seen as projection of $$x_k$$ onto column space of $$X_{(k)}$$, we can say:

$$X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k = \hat{x_k}$$, which is the regression prediction after regressing $$x_k$$ on $$X_{(k)}$$.

So our diagonal value simplifies to:

$$\sigma^2 [x_k^Tx_k - x_k^T X_{(k)} (X_{(k)}^TX_{(k)})^{-1}X_{(k)}^T x_k ] ^ {-1}= \sigma^2 [ x_k^Tx_k - x_k^T \hat{x_k} ] ^ {-1}$$

Now, since (I will omit subscript $$i$$ so $$x_k = x_{ik}$$)

$$R^2_k = 1 - \dfrac{ \sum (x_k - \hat{x_k} )^2 }{ \sum (x_k - \bar{x_k} )^2 }$$

$$\dfrac{\sigma^2}{(1-R^2_k) \sum (x_{k} - \bar{x_k})^2 }$$

will simplify to:

$$\dfrac{\sigma^2}{ \sum (x_{k} - \hat{x_k})^2 }$$

So in summary, we want to show that:

$$\sigma^2 [ x_k^Tx_k - x_k^T \hat{x_k} ] ^ {-1} = \dfrac{\sigma^2}{ \sum (x_{k} - \hat{x_k})^2 }$$

But then,

$$\sigma^2 [ x_k^Tx_k - x_k^T \hat{x_k} ] ^ {-1} = \dfrac{\sigma^2}{\sum (x_k)^2 - \sum x_k \hat{x_k} }$$

where as,

$$\dfrac{\sigma^2}{ \sum (x_{k} - \hat{x_k})^2 } = \dfrac{\sigma^2}{\sum (x_k)^2 - 2\sum x_k \hat{x_k} + \sum \hat{x_k}^2 }$$

Could someone please help me find where I am making a mistake, and if this is not the right approach, help me derive it correctly?

Resource A: http://people.stern.nyu.edu/wgreene/MathStat/GreeneChapter4.pdf (page 40)

It comes down to proving $$\sum x_k \hat{x_k} = \sum \hat{x_k} \hat{x_k}$$
We can see this because $$x_k = \hat{x_k} + e_k$$, where $$e_k$$ is the residual.
$$\sum x_k \hat{x_k} = \sum (\hat{x_k} + e_k) \hat{x_k}$$
$$= \sum \hat{x_k} \hat{x_k} + \sum e_k \hat{x_k}$$
But then $$\sum e_k \hat{x_k} = 0$$ since fitted values and residuals are orthogonal.