I have already read How to derive variance-covariance matrix of coefficients in linear regression.

Assume we're working with the usual simple regression model, $\mathbf{Y} \in \mathbb{R}^N$, $X \in M_{N \times (p+1)}(\mathbb{R})$, $\boldsymbol{\beta} \in \mathbb{R}^{p+1}$.

So, we have $$\text{Var}[\boldsymbol{\hat{\beta}}] = \text{Var}\left[ (X^{T}X)^{-1}X^{T}\mathbf{Y}\right] = (X^{T}X)^{-1}X^{T}\text{Var}[\mathbf{Y}]\left[(X^{T}X)^{-1}X^{T}\right]^{T}\text{.}$$ Now, $$\mathbf{Y} = X\boldsymbol{\beta}+\boldsymbol{\epsilon}$$ so $$\text{Var}\left[\mathbf{Y} \right] = \text{Var}\left[\boldsymbol{\epsilon}\right] = \sigma^2I_{N \times N}\text{.}$$ Thus, $$\text{Var}[\boldsymbol{\hat{\beta}}] = (X^{T}X)^{-1}X^{T}\sigma^2I_{N \times N}X[(X^{T}X)^{-1}]^{T}$$ and since $X^{T}X$ is symmetric, it follows that its inverse is symmetric, so $$\text{Var}[\boldsymbol{\hat{\beta}}] = (X^{T}X)^{-1}X^{T}\sigma^2I_{N \times N}X(X^{T}X)^{-1}\text{.}$$ I'm not sure how to proceed from here.

  • $\begingroup$ $\sigma^2$ is a scalar, why don't you take it out in front? $\endgroup$ – JohnK Dec 10 '15 at 15:25
  • $\begingroup$ @JohnK Yeah, I've figured it out now. $I_{N \times N}$ goes away, you end up with $\sigma^2 (X^{T}X)^{-1}X^{T}X(X^{T}X)^{-1} = \sigma^2(X^{T}X)^{-1}$. $\endgroup$ – Clarinetist Dec 10 '15 at 15:26

Easier than I thought. Any matrix times the identity gives the original matrix, so we have $$\begin{align} \text{Var}[\boldsymbol{\hat{\beta}}] &= (X^{T}X)^{-1}X^{T}\sigma^2I_{N \times N}X(X^{T}X)^{-1} \\ &= \sigma^2 (X^{T}X)^{-1}X^{T}I_{N \times N}X(X^{T}X)^{-1}\text{ since }\sigma^2\text{ is a constant} \\ &= \sigma^2 (X^{T}X)^{-1}X^{T}X(X^{T}X)^{-1}\\ &= \sigma^{2}(X^{T}X)^{-1}I_{(p+1)\times (p+1)} \\ &= \sigma^2(X^{T}X)^{-1}\text{.} \end{align}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.