Covariance matrix of least squares estimator $\hat{\beta}$ [duplicate]

Question

I read that the $Cov(\hat{\beta})=\sigma^2(Z'Z)^{-1}$, where $\hat{\beta}=(Z'Z)^{-1}Z'y$. However, I have yet been unable to find a proof of this fact online. Could anyone please provide a proof and/or a reference?

Answer 1

A good reference is Greene, Econometric Analysis. You should be able to pick up an older version (sixth edition or before) online for relatively cheap. Seventh is not noticeably better than sixth. I am changing your notation $Cov(\hat{\beta})$ to $V(\hat{\beta}_{\textrm{OLS}})$, but I mean the same thing by it.

Here is the proof:

If

1. $Y=Z\beta+\epsilon$
2. $E\left\{\epsilon|Z \right\}=0$
3. $V\left(\epsilon|Z \right)=\sigma^2I$
4. The OLS estimator exists and is unique (i.e. $Z'Z$ invertible)

then the OLS estimator is unbiased for $\beta$ and $V\left(\hat{\beta}_{\textrm{OLS}}|Z \right)=\sigma^2(Z'Z)^{-1}$.

Proof: Using the definition of the OLS estimator and then substituting in using 1: \begin{align} \hat{\beta}_{\textrm{OLS}} &= \left( Z'Z\right)^{-1}Z'Y\\ &= \left( Z'Z\right)^{-1}Z'\left( Z\beta+\epsilon \right)\\ &= \left( Z'Z\right)^{-1}Z'Z\beta + \left( Z'Z\right)^{-1}Z'\epsilon\\ &= \beta + \left( Z'Z\right)^{-1}Z'\epsilon \end{align} Taking expectations of both sides conditional on $Z$ gives you that the OLS estimator is unbiased. Taking variances on both sides conditional on $Z$ gives you: \begin{align} V\left( \hat{\beta}_{\textrm{OLS}} | Z \right) &= V\left( \beta + \left( Z'Z\right)^{-1}Z'\epsilon | Z \right)\\ &= V\left(\left( Z'Z\right)^{-1}Z'\epsilon | Z \right) \\ &= \left( Z'Z\right)^{-1}Z'V\left(\epsilon | Z \right) Z \left( Z'Z\right)^{-1} \\ &= \left( Z'Z\right)^{-1}Z'\sigma^2I Z \left( Z'Z\right)^{-1} \\ &= \sigma^2\left( Z'Z\right)^{-1}Z'Z \left( Z'Z\right)^{-1} \\ &= \sigma^2\left( Z'Z\right)^{-1} \end{align} QED

This does not quite give you what you asked for, since the variance is conditional on $Z$ rather than unconditional. If you want the variance to be unconditional, you have to additionally assume that $Z$ is fixed, so that the conditional variance becomes just an unconditional variance. On the other hand, this is the right variance conditional on the dataset you used to estimate $\beta$ with OLS, and inference based on this variance gives (asymptotically, if you don't assume $\epsilon$ normal) you correctly-sized hypothesis tests and confidence intervals.

Answer 2

This is the expression for the conditional variance-covariance matrix of the estimator. For the model $$Y=Z\beta + U, \; E(U\mid Z) =0,\; E(UU'\mid Z) = \sigma^2I$$ we have $$\operatorname {Cov}(\hat\beta \mid Z)=\operatorname {Cov} \left[(Z'Z)^{-1}Z'y\mid Z\right]$$

$$=\operatorname {Cov} \left[(Z'Z)^{-1}Z'(Z\beta +U)\mid Z\right] = \operatorname {Cov} \left[\beta +(Z'Z)^{-1}Z'U)\mid Z\right] = \operatorname {Cov} \left[(Z'Z)^{-1}Z'U)\mid Z\right] $$

Since $\beta$ is treated as a constant in the frequentist approach. Now

$$\operatorname {Cov} \left[(Z'Z)^{-1}Z'U)\mid Z\right] = E\Big\{\left[(Z'Z)^{-1}Z'U\right]\left[(Z'Z)^{-1}Z'U)\right]'\mid Z\Big\} - E\left[(Z'Z)^{-1}Z'U)\mid Z\right]E\left[(Z'Z)^{-1}Z'U)\mid Z\right]'$$

Since $$E\left[(Z'Z)^{-1}Z'U)\mid Z\right]' = (Z'Z)^{-1}Z'E\left[U\mid Z\right]' = 0$$ we are left with

$$\operatorname {Cov} \left[(Z'Z)^{-1}Z'U)\mid Z\right] = E\Big\{\left[(Z'Z)^{-1}Z'U\right]\left[(Z'Z)^{-1}Z'U)\right]'\mid Z\Big\} $$

$$= (Z'Z)^{-1}Z'E(UU'\mid Z)Z(Z'Z)^{-1}= (Z'Z)^{-1}Z'\sigma^2IZ(Z'Z)^{-1} $$

$$=\sigma^2(Z'Z)^{-1} $$