0
$\begingroup$

I am trying to prove $E(\hat{\beta} '\hat{\beta}) = \beta'\beta+\sigma^2 *\sum_{k=1}^K\lambda_k^{-1}$ where $\lambda_k$ denotes the eigenvalues of the matrix $(X'X)$ with dimensions $K\times K$. $\hat\beta$ is the least squares estimator for the regression $Y=X\beta +\epsilon$.

What I have so far is the following:

$E(\hat{\beta} '\hat{\beta})\\= E((X'X)^{-1}X'\epsilon +\beta)'((X'X)^{-1}X'\epsilon +\beta)) \\= \beta'\beta + E((X'\epsilon)'(X'X)^{-1}(X'X)^{-1}(X'\epsilon)) + E((X'\epsilon)'(X'X)^{-1}\beta) + E(\beta'(X'X)^{-1}(X'\epsilon))$

From the other end, this is how far I have come:

$\beta'\beta +\sigma^2 *\sum_{k=1}^K\lambda_k^{-1} \\=\beta'\beta+\sigma^2*\dfrac{1}{tr\{(X'X)\}} \\= \beta'\beta+ MSE(\hat\beta) \\=\beta'\beta + E((\hat\beta-\beta)'(\hat\beta-\beta))\\=\beta'\beta+E((X'\epsilon)'(X'X)^{-1}(X'X)^{-1}(X'\epsilon))$

Now to me it is entirely unclear, why the last 2 terms of the first equation should be 0? Did I make a mistake while transforming the equations or is this how it's supposed to be - and if so, why?

Any help is highly appreciated!

Improve this question
$\endgroup$
7
  • $\begingroup$ If $\hat{\beta}$ is an estimate of $\beta$, why are you trying to find the its estimate again? Once you estimate $\beta$, it become deterministic, so there won't be any covariance term. $\endgroup$
    – Maxtron
    Commented Oct 12, 2018 at 22:10
  • $\begingroup$ What do you mean exactly by that? I mean it's not about finding the estimate, but about the expected value of squared estimator? $\endgroup$
    – cabeer
    Commented Oct 12, 2018 at 22:37
  • 2
    $\begingroup$ $E(\beta'(X'X)^{-1}(X'\epsilon)) = 0$, because $E(\epsilon) = 0$ $\endgroup$
    – user158565
    Commented Oct 12, 2018 at 22:56
  • $\begingroup$ @a_statistician Oh damn, that does seem to make a lot of sense... But how come we can pull $\epsilon$ out of $E(.)$? Isn't this only allowed for scalar values? $\endgroup$
    – cabeer
    Commented Oct 12, 2018 at 23:03
  • 2
    $\begingroup$ In $\beta'(X'X)^{-1}(X'\epsilon)$, only $\epsilon$ is random, $X$ and $\beta$ are constant. So $E(\beta'(X'X)^{-1}(X'\epsilon)) = \beta'(X'X)^{-1}X'E(\epsilon) = \beta'(X'X)^{-1}X'0 = 0$ $\endgroup$
    – user158565
    Commented Oct 13, 2018 at 0:21

1 Answer 1

Reset to default
1
$\begingroup$

We need following:

  1. $\beta$ is unknown constant vector, regression coefficients.

  2. $X$ is constant design matrix.

  3. $E(\epsilon) = 0$ from the assumptions.

  4. $E(AY)=AE(Y)$ for constant matrix $A$ and random vector $Y$.

Then we have $$E(β′(X′X)^{−1}(X′ϵ))=β′(X′X)^{−1}X′E(ϵ)=β′(X′X)^{−1}X′0=0 $$

Improve this answer
$\endgroup$
3
  • $\begingroup$ Did you perhaps lose part of your post? Your result does not appear to answer any part of the question--it only seems like some kind of a preliminary to an answer. $\endgroup$
    – whuber ♦
    Commented Oct 13, 2018 at 20:26
  • $\begingroup$ @whuber The question is "why the last 2 terms of the first equation should be 0?" $\endgroup$
    – user158565
    Commented Oct 13, 2018 at 21:04
  • $\begingroup$ Fair enough (+1). I had interpreted the post in the sense of the first implied question; namely, finding an expression for the expectation of the squared norm of the estimated coefficients. Because there are important flaws in the derivation that is presented, the two questions are not the same! $\endgroup$
    – whuber ♦
    Commented Oct 14, 2018 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.