4
$\begingroup$

I was reading from The Elements of Statistical Learning (Section 3.2) and they were able to find $Var(\hat\beta)$

Given

$$\hat\beta = (X^TX)^{-1}X^Ty \qquad (3.6) $$

The answer they had was $Var(\hat\beta) = (X^TX)^{-1}\sigma^2$

Please how do I find $Var(\hat\beta)$ from $ (3.6)$

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
6
$\begingroup$

Via substitution and some matrix algebra: $$\begin{align}\operatorname{var}(\hat\beta)&=\mathbb E[\hat\beta\hat\beta^T]-\mathbb E[\hat\beta]\mathbb E[\hat \beta^T]\\&=\mathbb E[(X^TX)^{-1}X^Tyy^TX(X^TX)^{-1}]-\mathbb \beta\beta^T\\&=(X^TX)^{-1}X^T\mathbb E[yy^T]X(X^TX)^{-1}-\beta\beta^T\\&=(X^TX)^{-1}X^T(\sigma^2I+X\beta\beta^TX^T)X(X^TX)^{-1}-\beta\beta^T\\&=\sigma^2(X^TX)^{-1}\end{align}$$

Note that since $y=X\beta+\epsilon$, and $\mathbb E[\epsilon\epsilon^T]=\sigma^2I$ and $\mathbb E[\epsilon]=0$: $$\mathbb E[yy^T]=X\beta \beta^TX+\mathbb E[\epsilon\epsilon^T]=X\beta \beta^TX+\sigma^2I$$

Improve this answer
$\endgroup$
8
  • 4
    $\begingroup$ The algebra would become less tedious if using the formula for the variance matrix of linear transformations of random vectors $\operatorname{Var}(Ay)=A\operatorname{Var}(y)A^T$ $\endgroup$
    – Jarle Tufto
    Commented Oct 23, 2020 at 18:50
  • $\begingroup$ @gunnes does it mean $var(\hat\beta) = E[\hat\beta\hat\beta^T] - E[\hat\beta]E[\hat\beta^T]$ $\endgroup$
    – EA Lehn
    Commented Oct 23, 2020 at 19:09
  • $\begingroup$ @EALehn yes, that is the definition of covariance for random vectors $\endgroup$
    – gunes
    Commented Oct 23, 2020 at 19:11
  • 1
    $\begingroup$ @gunnes why is the $E[(X^TX)^{-1}X^Ty] = (X^TX)^{-1}X^T E[y] $ $\endgroup$
    – EA Lehn
    Commented Oct 23, 2020 at 19:29
  • 2
    $\begingroup$ Because only randomness is inside $y$ $\endgroup$
    – gunes
    Commented Oct 23, 2020 at 19:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.