Derivation of standard error of regression coefficients

Question

I know that variance of the regression estimate is given by $\widehat{\textrm{Var}}(\hat{\mathbf{\beta}}) = \hat{\sigma}^2 (\mathbf{X}^{\prime} \mathbf{X})^{-1}$. However, in many places, it is given that

A mathjax form: $$\sqrt{\frac {S_{y,1,2..k}^2} {\Sigma x_i^2(1-R_{1,2,..k}^2)}}$$ where $R_{1,2,..k}^2$ is the $R^2$ is the regression value when $x_i$ is regressed against all the remaining independent variables.

How do we simplify to the form shown above from $\hat{\sigma}^2 (\mathbf{X}^{\prime} \mathbf{X})^{-1}$

Here is my attempt:

$$X'X = \begin{bmatrix}A & B \\C & D \end{bmatrix}$$ Where, $$\begin{align*} A &= \mathbf{x_1}'\mathbf{x_1}\quad \quad &\text{1 by 1 matrix}\\ B &= \mathbf{x_1}'X_{-1} \quad &\text{1 by n-1 matrix}\\ C &= X_{-1}\mathbf{x_1} & \text{n-1 by 1 matrix} \\ D &= X_{-1}'X_{-1} & \text{n-1 by n-1 matrix} \end{align*}$$

Using Schur complement, we can get that $$\left(X'X \right)^{-1} = \begin{bmatrix}\left(A - BD^{-1}C \right)^{-1} & \ldots \\ \ldots & \ldots \end{bmatrix}$$

It is straightforward to see from here that: $A$ in $(A - BD^{-1}C)^{-1}$ is $\Sigma x_i^2$ and $S_{y,1,2..k}^2 = \sigma^2$. Can you please help with simplifying the remaining terms.

Answer 1

Write $X = (Z \vert W)$ where $Z$ is the first column and $W$ is the $n \times p-1$ matrix with the remaining predictors. I'm borrowing a lot from this answer of mine.

We have that $$ Var(\hat \beta_Z) = \sigma^2 (X^TX)^{-1}_{11} $$ so we only need to work out $(X^TX)^{-1}_{11}$ and can change which column of $X$ we make $Z$ to get the result for any other coefficient.

Now we have $$ X^TX = \left[ \begin{array}{c|c} Z^TZ & Z^TW \\ \hline W^TZ & W^TW \end{array} \right] $$ and by assumption $X$ is full column rank so $X^TX$ is invertible, and this means that $W$ is full column rank too so $W^TW$ is also invertible.

Let $H = W(W^TW)^{-1}W^T$ be the projection matrix onto the column space of $W$. Then by the inverse of a block diagonal matrix we have $$ (X^TX)^{-1} = \left[ \begin{array}{c|c} (Z^T(I-H)Z)^{-1} & B \\ \hline C & D \end{array} \right] $$ where $B$, $C$, and $D$ don't matter. This means $$ Var(\hat \beta_Z) = \frac{\sigma^2}{Z^T(I-H)Z}. $$

Now suppose that our predictors are standardized so that $\bar Z = 0$ (the result does not necessarily hold if this isn't the case). This means that $$ 1-R^2_Z = \frac{SS^{(Z)}_{res}}{SS_{tot}^{(Z)}} = \frac{Z^T(I-H)Z}{Z^TZ} $$ so we can write $$ Var(\hat \beta_Z) = \frac{\sigma^2}{Z^T(I-H)Z} = \frac{\sigma^2}{Z^TZ (1-R_Z^2)} = \frac{\sigma^2}{\sum_{i=1}^n z_i^2(1-R_z^2)} $$ as desired.