I have read many of similar posts here already as well as other resources on the topic, but they all generally just show the steps that generate this equation:


What I want to know is how the above matrix operations work under the hood. How do they generate something like the equation below for each slope coefficient in a regression? How is the variance-covariance matrix being generated by those matrix operations?

$\sigma^{2}=\frac{\sum ( X - \mu )^2}{N}$

I look at the above equation and it makes sense to me what is happening, but I am having trouble seeing the matrix operation in the same way and it is annoying me.

I am by no means strong in matrix alebra and I realize that there may be something fundamental I just don't know.

I understand this is a big task, but any help is appreciated.

  • 1
    $\begingroup$ Your first formula isn't an equation at all and the second one is not a matrix formula, so it is difficult to determine what you are asking. Probably there are typographical errors: could you please edit the post to fix them? $\endgroup$ – whuber Apr 1 '15 at 18:39
  • $\begingroup$ Do you mean $\hat\sigma^2$ in your first mathematical expression? $\endgroup$ – Michael M Apr 1 '15 at 18:57
  • $\begingroup$ @Michael M, yes I do. Fixed. $\endgroup$ – Michael Apr 1 '15 at 19:02
  • $\begingroup$ The basic properties of expectation and variance like $E(AX)=AE(X)$ and $\text{Var}(AX)=A \text{Var}(X)A'$ can be verified element-by-element from the univariate properties of expectation, variance and covariance. $\endgroup$ – Glen_b Apr 2 '15 at 0:55

My best short, intuitive explanation:

Take that $X'X$. That is like the covariance of the $X$'s (it is actually the sum of squares). So, the variance (ish) of each $X$, and how each one changes when the other is at a certain value.

Take that $\hat\sigma$. That is the residual variance. On average, how far does a point lie from its estimated expected value? Since it is scaled by $n$, think of it like a standard error. It is the uncertainty in the expectation of $Y$ given $X$.

Now, scale that ``standard error'' by the "covariance" (actually sum of squares) of the data. What you have is the variance in the expectation of $Y$ per unit variance (covariance) in the data. This is an interpretation of the variance of $\hat\beta$.

All that said, understanding the formalism (including an understanding of why the derivation is structured the way it is) will help with the intuition more than an intuitive explanation will help with the intuition.

  • 1
    $\begingroup$ Unless you first center the columns of $X$ at $0$, $X^\prime X$ is not the covariance matrix. Even when you do center the columns, it is only a fixed multiple of the covariance matrix. A fairly standard name for it is the sum-of-squares matrix. $\endgroup$ – whuber Apr 1 '15 at 18:36
  • $\begingroup$ Right, I was being sloppy. Thanks. Fixed. I think I gave a reasonably intuitive explanation though. $\endgroup$ – generic_user Apr 1 '15 at 18:50
  • $\begingroup$ But why does inverting the X′X matrix and multiplying by the variance of the residuals produce the variance-covariance matrix that we eventually get the standard errors of the residuals from? What is the algebra happening underneath the hood? $\endgroup$ – Michael Apr 2 '15 at 2:01
  • $\begingroup$ I don't know what you mean by "under the hood". What does 2+2 mean under the hood? $\endgroup$ – generic_user Apr 2 '15 at 5:22
  • 2
    $\begingroup$ You can actually say quite a lot about what $2+2$ means under the hood! : ) en.wikipedia.org/wiki/Principia_Mathematica $\endgroup$ – Matthew Drury May 25 '15 at 17:03

The variance is defined as: $Var(X) = E[(X-\mu)^2]$. For OLS we can write it as:

\begin{align} E[(\hat{\beta} - \beta)(\hat{\beta} - \beta)'] &= E[((X’X)^{-1}X’u)((X’X)^{-1}X’u)’] \\ &= E[(X’X)^{-1}X’uu'X(X’X)^{-1}] \\ \text{Now assume fixed X, and we can write:} \\ &= (X’X)^{-1}X’E[uu’]X(X’X)^{-1} \\ \text{Remember that: } E[uu’] = \sigma^2I \\ &= (X’X)^{-1}X’(\sigma^2I)X(X’X)^{-1} \\ &= \sigma^2I(X’X)^{-1}X’X(X’X)^{-1} \\ &= \sigma^2(X'X)^{-1} \end{align}

It is then easy to show that the estimator is unbiased, consistent and efficient. Also you could estimate $\hat{\sigma}^2 = \frac{uu'}{n-k-1}$, which you would do in practice.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.