0
$\begingroup$

I am trying to Derive the Estimated Variance of Regression Coefficients. I am struggling with the algebra.

Here is the model:

$$y = X\beta + \epsilon$$ $$\hat{\beta} = (X^TX)^{-1}X^Ty$$

First, we find out the expectation:

$$E(\hat{\beta}) = E((X^TX)^{-1}X^Ty) = E((X^TX)^{-1}X^T(X\beta + \epsilon)) = \beta + (X^TX)^{-1}X^TE(\epsilon)$$

Next, we find out the squared expectation:

$$E(\hat{\beta}^2) = E((X^TX)^{-1}X^Ty((X^TX)^{-1}X^Ty)^T) = E((X^TX)^{-1}X^Tyy^TX(X^TX)^{-1})$$

Substituting $y = X\beta + \epsilon$ gives:

$$E(\hat{\beta}^2) = E((X^TX)^{-1}X^T(X\beta + \epsilon)(X\beta + \epsilon)^TX(X^TX)^{-1}) = \beta\beta^T + E((X^TX)^{-1}X^T\epsilon\epsilon^TX(X^TX)^{-1})$$

Assuming that the errors are uncorrelated and homoscedastic, this simplifies to:

$$E(\hat{\beta}^2) = \beta\beta^T + \sigma^2(X^TX)^{-1}$$

Finally, we can evaluate Var($\hat{\beta}$) = E($\hat{\beta}^2$) - E($\hat{\beta}$)^2 :

$$\text{Var}(\hat{\beta}) = \beta\beta^T + \sigma^2(X^TX)^{-1} - \beta^2 = \sigma^2(X^TX)^{-1}$$

Is this the correct derivation?

Improve this question
$\endgroup$
6
  • $\begingroup$ Seems good for the most part. I would replace $\beta^2$ and $\hat{\beta}^2$ with $\beta\beta^\top$ and $\hat{\beta} \hat{\beta}^\top$ though. $\endgroup$
    – angryavian
    Commented Mar 13 at 4:30
  • 2
    $\begingroup$ What do you mean by beta-regression? Your model seems to be ordinary least squares regression. Beta regression is about modeling a beta distributed variable. $\endgroup$
    – Sextus Empiricus
    Commented Mar 13 at 8:36
  • $\begingroup$ "Assuming that the errors are uncorrelated and homoscedastic" -- for beta regression this is, in general, not the case. $\endgroup$
    – wzbillings
    Commented Mar 13 at 14:10
  • $\begingroup$ sorry i meant OLS ... the beta coefficients $\endgroup$
    – user409209
    Commented Mar 13 at 14:59
  • 1
    $\begingroup$ You made it too verbose -- it is a straightforward application of $\mathrm{Var}(A\xi) = A\mathrm{Var}(\xi)A^\top$ for non-random $A$ and random vector $\xi$. What you did is like rediscovering this well-known formula for this special case. $\endgroup$
    – Zhanxiong
    Commented Mar 13 at 16:04

0

Reset to default