10
$\begingroup$

I stumbled over a rather simple result of OLS regression which is

$$ Var[\hat\beta] = \sigma^2(X^TX)^{-1} $$ where $\sigma$ is the variance of the error term $u$ and $X$ is the regressor matrix.

I first just accepted the proof in my textbook but now I am thinking that it either uses sloppy notation or I am missing something. $\hat\beta$ is the estimated and $\beta$ is the true parameter (assuming unbiasedness).

It states that

\begin{align} Var[\hat\beta] &= E[(\hat\beta - \beta)(\hat\beta-\beta)^T] \\ &= E[(X^TX)^{-1}X^Tuu^TX(X^TX)^{-1}] \\ &= (X^TX)^{-1}X^T E[uu^T] X(X^TX)^{-1} \end{align}

but $X$ was only assumed to be exogenous and not non-stochastic. Under this assumption I think $X$ cannot be dragged outside the expectation operator.

Momentarily, I think that it should be $Var[\hat\beta|X]$ to make sense. Is that the case? My web research couldn't clarify this. I only found similar derivations to the above without further explainations.

Improve this question
$\endgroup$
5
  • $\begingroup$ This looks like frequentist statistics and thus data or observations should be random, not the inputs or regressors, shouldn't they? The data are random because there is random error added. $\endgroup$
    – gwr
    Nov 28, 2015 at 11:40
  • $\begingroup$ A couple of pages before this derivation the book authors state that assuming $X$ as nonstochastic is "frequently not a reasonable assumption" . Therfore, this assumption was relaxed to exogeneity. $\endgroup$
    – Alex
    Nov 28, 2015 at 11:46
  • 1
    $\begingroup$ In section 6 of this tutorial it gives that the simplification only holds if $X$ is assumed non-stochastic. $\endgroup$
    – gwr
    Nov 28, 2015 at 12:03
  • $\begingroup$ This would support my assumption that if we assume only exogeneity the correct notation would have been $Var[\hat\beta|X]$ and not $Var[\hat\beta]$. But why is this fact skipped so often? Seems not correct to me. $\endgroup$
    – Alex
    Nov 28, 2015 at 13:42
  • 1
    $\begingroup$ It seems that under the assumptions usually made the usual estimate for the variance given that $X$ is fixed is also an unbiased estimate of the unconditional estimate when $X$ is random as is shown here. Maybe that helps. $\endgroup$
    – gwr
    Nov 28, 2015 at 15:14

1 Answer 1

Reset to default
6
$\begingroup$

You are right that the conditional variance is not generally the same as the unconditional one. By the variance decomposition lemma, which says that, for r.v.s $X$ and $Y$

$$ Var(X)=E[Var(X|Y)]+Var[E(X|Y)] $$ Translated to our problem, $$ Var(\widehat{\beta})=E[Var(\widehat{\beta}|X)]+Var[E(\widehat{\beta}|X)] $$ Now, using that OLS is conditionally unbiased (under suitable assumptions like exogeneity assumed here), we have $$ E(\widehat{\beta}|X)=\beta $$ and thus $$ Var[E(\widehat{\beta}|X)]=0,$$ as $\beta$ is a constant, so that $$ Var(\widehat{\beta})=E[Var(\widehat{\beta}|X)] $$ or $$Var(\widehat{\beta})=E[\sigma^2(X'X)^{-1}]=\sigma^2E[(X'X)^{-1}].$$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Oh, just realized that this is pretty much what the link of @gwr says... $\endgroup$
    – Christoph Hanck
    Nov 28, 2015 at 16:37
  • 1
    $\begingroup$ +1 I am only a biased Bayesian and thus have forgotten long ago about what unbiased estimators really are good for and thus had to use secondary material to answer anyways. As a pun: Your concise answer is better for the long run... :-) $\endgroup$
    – gwr
    Nov 28, 2015 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.