I know that variance of the regression estimate is given by $\widehat{\textrm{Var}}(\hat{\mathbf{\beta}}) = \hat{\sigma}^2 (\mathbf{X}^{\prime} \mathbf{X})^{-1}$. However, in many places, it is given that

enter image description here

A mathjax form: $$\sqrt{\frac {S_{y,1,2..k}^2} {\Sigma x_i^2(1-R_{1,2,..k}^2)}}$$ where $R_{1,2,..k}^2$ is the $R^2$ is the regression value when $x_i$ is regressed against all the remaining independent variables.

How do we simplify to the form shown above from $\hat{\sigma}^2 (\mathbf{X}^{\prime} \mathbf{X})^{-1}$

Here is my attempt:

$$X'X = \begin{bmatrix}A & B \\C & D \end{bmatrix}$$ Where, $$\begin{align*} A &= \mathbf{x_1}'\mathbf{x_1}\quad \quad &\text{1 by 1 matrix}\\ B &= \mathbf{x_1}'X_{-1} \quad &\text{1 by n-1 matrix}\\ C &= X_{-1}\mathbf{x_1} & \text{n-1 by 1 matrix} \\ D &= X_{-1}'X_{-1} & \text{n-1 by n-1 matrix} \end{align*}$$

Using Schur complement, we can get that $$\left(X'X \right)^{-1} = \begin{bmatrix}\left(A - BD^{-1}C \right)^{-1} & \ldots \\ \ldots & \ldots \end{bmatrix}$$

It is straightforward to see from here that: $A$ in $(A - BD^{-1}C)^{-1}$ is $\Sigma x_i^2$ and $S_{y,1,2..k}^2 = \sigma^2$. Can you please help with simplifying the remaining terms.


Write $X = (Z \vert W)$ where $Z$ is the first column and $W$ is the $n \times p-1$ matrix with the remaining predictors. I'm borrowing a lot from this answer of mine.

We have that $$ Var(\hat \beta_Z) = \sigma^2 (X^TX)^{-1}_{11} $$ so we only need to work out $(X^TX)^{-1}_{11}$ and can change which column of $X$ we make $Z$ to get the result for any other coefficient.

Now we have $$ X^TX = \left[ \begin{array}{c|c} Z^TZ & Z^TW \\ \hline W^TZ & W^TW \end{array} \right] $$ and by assumption $X$ is full column rank so $X^TX$ is invertible, and this means that $W$ is full column rank too so $W^TW$ is also invertible.

Let $H = W(W^TW)^{-1}W^T$ be the projection matrix onto the column space of $W$. Then by the inverse of a block diagonal matrix we have $$ (X^TX)^{-1} = \left[ \begin{array}{c|c} (Z^T(I-H)Z)^{-1} & B \\ \hline C & D \end{array} \right] $$ where $B$, $C$, and $D$ don't matter. This means $$ Var(\hat \beta_Z) = \frac{\sigma^2}{Z^T(I-H)Z}. $$

Now suppose that our predictors are standardized so that $\bar Z = 0$ (the result does not necessarily hold if this isn't the case). This means that $$ 1-R^2_Z = \frac{SS^{(Z)}_{res}}{SS_{tot}^{(Z)}} = \frac{Z^T(I-H)Z}{Z^TZ} $$ so we can write $$ Var(\hat \beta_Z) = \frac{\sigma^2}{Z^T(I-H)Z} = \frac{\sigma^2}{Z^TZ (1-R_Z^2)} = \frac{\sigma^2}{\sum_{i=1}^n z_i^2(1-R_z^2)} $$ as desired.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.