What does the asymptotic heteroskedastic variance-covariance matrix of beta look like?

Question

I don't really understand the variance-covariance matrix of beta when there is heteroscedasticity. Can someone explain what the matrix looks like? I am talking about:
$$V_\beta=Q_{xx}^{-1}\Omega Q_{xx}^{-1}$$

where,

• $V_\beta$ is the variance of beta (asymptotic, so equal to the population variance for large n).
• $Q_{xx}$ is $X'X$ and $\Omega$ is the variance of $X'e$ where the estimator for $\beta$ is $(X'X)^{-1}X'y$ and $e=y-X\beta$. Therefore $y=e+X\beta$ is substituted for $Y$ and then solved for $\hat\beta-\beta$, getting $\hat\beta-\beta=(X'X)^{-1}X'e$.

So if I understand correctly, the variance-covariance matrix shows the variance on the diagonal of $\beta$, but what are on the off diagonal elements?

Answer 1

Ok, so start with $$\hat\beta = (X^TX)^{-1}X^TY$$ and write this as $\hat\beta=AY$.

We now have $$\mathrm{var}[\hat\beta] = A^T\mathrm{var}[Y]A$$ as a property of linear transformations, so $$\mathrm{var}[\hat\beta] = (X^TX)^{-1} X^T\mathrm{var}[Y]X (X^TX)^{-1}$$ You have this re-grouped as $$\mathrm{var}[\hat\beta] = (X^TX)^{-1} \left(X^T\mathrm{var}[Y]X\right) (X^TX)^{-1}$$

So, what does the matrix $\mathrm{var}[Y]$ look like? If different $Y$s are independent, the matrix is diagonal, with the variances of individual $Y$s as the diagonal elements. $\Omega$ doesn't have any very illuminating form; it's a sum of matrices of the form $x_ix_i^T$ weighted by the individual variances of the observations.

If the variances are all the same, $\mathrm{var}[Y]$ is just $\sigma^2$ times an $n\times n$ identity matrix and the formula simplifies to the familiar homoscedastic formula. $\Omega$ is then just $\sigma^2 X^TX$.

Asymptotically, it's convenient to put in some $n$s and consider $$n\mathrm{var}[\hat\beta]= \left(\frac{1}{n} X^TX)\right)^{-1} \left(\frac{1}{n}X^T\mathrm{var}[Y]X\right)\left(\frac{1}{n} X^TX\right)^{-1}$$ This has the advantage that all the bits are means and actually converge to non-zero constants (under some assumptions) with $ n\mathrm{var}[\hat\beta]$ being the limiting variance of $\sqrt{n}(\hat\beta-\beta_0)$. By contrast $\mathrm{var}[\hat\beta]$ just converges to zero; $\mathrm{\hat\beta}$ gets less and less variable as $n$ increases.

Finally, you might also be interested in the heteroscedasticity-consistent estimator that's a plug-in version of this. The estimator has the same $X^TX$ matrix $$\widehat{\mathrm{var}}[\hat\beta] = (X^TX)^{-1} \left(X^T\widehat{\mathrm{var}}[Y]X\right) (X^TX)^{-1}$$ Here, $\widehat{\mathrm{var}}[Y]$ is again a diagonal matrix, with the diagonal elements being $(y_i-x_i\hat\beta)^2$. Again, if you want to work asymptotically it's convenient to put in $1/n$s everywhere so that the components of the estimator are each means that have straightforward limits.

Answer 2

As tchakravarty noticed, $\beta$ is a column vector of all the parameters of the model. So the variance-covariance matrix shows on the diagonals the covariance of the $\beta$ with itself (=variance), and on the off diagonal elements it shows the covariance of individual $\beta$'s with other $\beta$'s.

In conclusion shows this matrix the asymptotic heteroskedastic variance of the estimator $\hat \beta$, which converges by the Central Limit Theorem (CLT) and the Weak Law of Large Numbers (WLLN) to the variance of the population $\beta$.