Covariance matrix of least squares estimator $\hat{\beta}$ [duplicate]

I read that the $Cov(\hat{\beta})=\sigma^2(Z'Z)^{-1}$, where $\hat{\beta}=(Z'Z)^{-1}Z'y$. However, I have yet been unable to find a proof of this fact online. Could anyone please provide a proof and/or a reference?

A good reference is Greene, Econometric Analysis. You should be able to pick up an older version (sixth edition or before) online for relatively cheap. Seventh is not noticeably better than sixth. I am changing your notation $Cov(\hat{\beta})$ to $V(\hat{\beta}_{\textrm{OLS}})$, but I mean the same thing by it.

Here is the proof:

If

1. $Y=Z\beta+\epsilon$
2. $E\left\{\epsilon|Z \right\}=0$
3. $V\left(\epsilon|Z \right)=\sigma^2I$
4. The OLS estimator exists and is unique (i.e. $Z'Z$ invertible)

then the OLS estimator is unbiased for $\beta$ and $V\left(\hat{\beta}_{\textrm{OLS}}|Z \right)=\sigma^2(Z'Z)^{-1}$.

Proof: Using the definition of the OLS estimator and then substituting in using 1: \begin{align} \hat{\beta}_{\textrm{OLS}} &= \left( Z'Z\right)^{-1}Z'Y\\ &= \left( Z'Z\right)^{-1}Z'\left( Z\beta+\epsilon \right)\\ &= \left( Z'Z\right)^{-1}Z'Z\beta + \left( Z'Z\right)^{-1}Z'\epsilon\\ &= \beta + \left( Z'Z\right)^{-1}Z'\epsilon \end{align} Taking expectations of both sides conditional on $Z$ gives you that the OLS estimator is unbiased. Taking variances on both sides conditional on $Z$ gives you: \begin{align} V\left( \hat{\beta}_{\textrm{OLS}} | Z \right) &= V\left( \beta + \left( Z'Z\right)^{-1}Z'\epsilon | Z \right)\\ &= V\left(\left( Z'Z\right)^{-1}Z'\epsilon | Z \right) \\ &= \left( Z'Z\right)^{-1}Z'V\left(\epsilon | Z \right) Z \left( Z'Z\right)^{-1} \\ &= \left( Z'Z\right)^{-1}Z'\sigma^2I Z \left( Z'Z\right)^{-1} \\ &= \sigma^2\left( Z'Z\right)^{-1}Z'Z \left( Z'Z\right)^{-1} \\ &= \sigma^2\left( Z'Z\right)^{-1} \end{align} QED

This does not quite give you what you asked for, since the variance is conditional on $Z$ rather than unconditional. If you want the variance to be unconditional, you have to additionally assume that $Z$ is fixed, so that the conditional variance becomes just an unconditional variance. On the other hand, this is the right variance conditional on the dataset you used to estimate $\beta$ with OLS, and inference based on this variance gives (asymptotically, if you don't assume $\epsilon$ normal) you correctly-sized hypothesis tests and confidence intervals.

• Thank you for your help. That was exactly what I was looking for. Could you please explain to me why you could treat $\beta$ as a constant (I'm referring to the second equality of the last block of derivations)?
– rbm
Oct 16, 2013 at 12:22
• $V(const + W) = Var(W)$ is a property of variance operator. Basically, constant terms do not appear since $V(const + W) = E[(const + w - E[const + W])^2] = E[(W - E[W])^2] = V(W)$. Oct 16, 2013 at 12:38
• @Theja Thanks but that was actually not what I meant. I know about that property, but what I tried to ask is why $\beta$ can be seen as a constant. Isn't it some kind of random variable?
– rbm
Oct 16, 2013 at 12:50
• $\beta$ is the true unknown model which is assumed to be fixed and not random (see point 1. in the answer above). Oct 16, 2013 at 12:55
• @rbm Well, if you are a Bayesian, then you are going to see that step as a mistake (I think---but ask a Bayesian to be sure). If you are a Frequentist, parameters are always constants.
– Bill
Oct 16, 2013 at 13:12

This is the expression for the conditional variance-covariance matrix of the estimator. For the model $$Y=Z\beta + U, \; E(U\mid Z) =0,\; E(UU'\mid Z) = \sigma^2I$$ we have $$\operatorname {Cov}(\hat\beta \mid Z)=\operatorname {Cov} \left[(Z'Z)^{-1}Z'y\mid Z\right]$$

$$=\operatorname {Cov} \left[(Z'Z)^{-1}Z'(Z\beta +U)\mid Z\right] = \operatorname {Cov} \left[\beta +(Z'Z)^{-1}Z'U)\mid Z\right] = \operatorname {Cov} \left[(Z'Z)^{-1}Z'U)\mid Z\right]$$

Since $\beta$ is treated as a constant in the frequentist approach. Now

$$\operatorname {Cov} \left[(Z'Z)^{-1}Z'U)\mid Z\right] = E\Big\{\left[(Z'Z)^{-1}Z'U\right]\left[(Z'Z)^{-1}Z'U)\right]'\mid Z\Big\} - E\left[(Z'Z)^{-1}Z'U)\mid Z\right]E\left[(Z'Z)^{-1}Z'U)\mid Z\right]'$$

Since $$E\left[(Z'Z)^{-1}Z'U)\mid Z\right]' = (Z'Z)^{-1}Z'E\left[U\mid Z\right]' = 0$$ we are left with

$$\operatorname {Cov} \left[(Z'Z)^{-1}Z'U)\mid Z\right] = E\Big\{\left[(Z'Z)^{-1}Z'U\right]\left[(Z'Z)^{-1}Z'U)\right]'\mid Z\Big\}$$

$$= (Z'Z)^{-1}Z'E(UU'\mid Z)Z(Z'Z)^{-1}= (Z'Z)^{-1}Z'\sigma^2IZ(Z'Z)^{-1}$$

$$=\sigma^2(Z'Z)^{-1}$$

• Could you explain why $E(UU'\mid Z) = \sigma^2I$ does that imply that the variance of each error term is equal to $\sigma^2$? I rather thought that it should be a diagnoal matrix. Which assumption is this (uncorrelated errors + Normal distribution?)? Jun 7, 2016 at 16:26
• @MrYouMath $I$ is the identity matrix and so it is diagonal. This is the "homoskedastic and uncorrelated errors" assumption. Normality is not assumed. Jun 7, 2016 at 16:49