5
$\begingroup$

I am trying to prove that in multivariate linear regression $MSE = (n-2)\sigma^2 $

Here is my approach:

Under the usual notation,

$$ Y = X\beta + \epsilon \\ $$ $$ \hat Y = X\hat\beta \\ $$ $$ \hat\beta = (X'X)^{-1}X'Y \\ \\ \implies \hat\beta' = Y'X(X'X)^{-1} $$

Now, \begin{align} \Sigma (Y_i - \hat Y_i)^2 & = (Y_i - \hat Y_i)'(Y_i - \hat Y_i) \\ & = (X(\beta - \hat \beta) + \epsilon)' (X(\beta - \hat \beta) + \epsilon)\\ & = \underbrace {(\beta - \hat \beta)'X'X(\beta - \hat \beta)}_{term1} + \underbrace {\epsilon'X (\beta - \hat \beta)}_{term2}\\ & + \underbrace {(\beta - \hat \beta)'X'\epsilon}_{term3} + \epsilon'\epsilon \\ \end{align}

Simplifying the individual terms

Term 1: \begin{align} (\beta - \hat \beta)'X'X(\beta - \hat \beta) &= (\beta - (X'X)^{-1}X'Y)'X'X(\beta - (X'X)^{-1}X'Y)\\ & = (\beta' - Y'X(X'X)^{-1})X'X(\beta - (X'X)^{-1}X'Y) \\ & = \beta'X'X\beta - Y'X\beta - \beta'(X'X)(X'X)^{-1}X'Y + Y'X(X'X)^{-1}X'Y \\ & = \beta'X'X\beta - (\beta'X' + \epsilon')X\beta - \beta'(X'X)(X'X)^{-1}X'Y + \\ & (\beta'X' + \epsilon')X(X'X)^{-1}X'Y \quad \text{(substituting the value of }Y') \\ & = - \epsilon'X\beta + \epsilon'X(X'X)^{-1}X'Y \quad \text {some terms get cancelled} \\ & = - \epsilon'X\beta + \epsilon'X(X'X)^{-1}X'( X\beta + \epsilon) \quad \text {substituting the value of } Y \\ & = \epsilon'X(X'X)^{-1}X'\epsilon \end{align}

Term 2 : \begin{align} \epsilon'X (\beta - \hat \beta) &= \epsilon'X(\beta - (X'X)^{-1}X'Y)\\ & = \epsilon'X(\beta - (X'X)^{-1}X'X\beta)\quad \text {substituting the value of } Y \\\\ & = 0 \end{align}

As Term 3 is transpose of Term 2, Term 3 = 0

\begin{align} \Sigma (Y_i - \hat Y_i)^2 & = \epsilon'X(X'X)^{-1}X'\epsilon + \epsilon'\epsilon \\ E(\Sigma (Y_i - \hat Y_i)^2) & = E(\epsilon'X(X'X)^{-1}X'\epsilon + \epsilon'\epsilon) \\ \end{align} I'm stuck here, unable to make any further simplifications. Can someone please help.

What further baffles me is the RHS term is greater than $n\sigma^2$ as $E(\epsilon'\epsilon) = n*\sigma$

$\endgroup$
  • 1
    $\begingroup$ I'd imagine that those 3 terms should be (together) negative since $\hat{\beta}$ minimizes the $\sum(Y_i-\hat{Y_i})^2$ and should thus be smaller or (at least equal) then the case $\hat{\beta}=\beta$ for which the 3 terms become zero. I wonder how you get to the term 2 and 3 being zero? another note: you might like these two links. Especially the proofs for the sample variance as unbiased estimator. I imagine it can be done analogous for the multivariate case (note: use n-p instead of n-2). en.wikipedia.org/wiki/Studentized_residual en.wikipedia.org/wiki/Bias_of_an_estimator $\endgroup$ – Sextus Empiricus Sep 11 '17 at 22:24
  • 1
    $\begingroup$ In the simplification of term 2 you substitute $Y$ with $X \beta$ instead of $X \beta + \epsilon$. So instead you got $\beta - \hat{\beta} = (X^\prime X)^{-1} X^\prime \epsilon$, and $X(\beta - \hat{\beta})$ is the projection of $\epsilon$ onto the span of $X$. $\endgroup$ – Sextus Empiricus Sep 11 '17 at 22:53
6
$\begingroup$

Martijn Weterings's commnet is very useful. Your derivation of term 2 is wrong.

$\epsilon'X (\beta - \hat \beta) \\= \epsilon'X(\beta - (X'X)^{-1}X'Y) \\=\epsilon'X\left\{\beta - (X'X)^{-1}X'(X\beta+\epsilon)\right\}\\=\epsilon'X \left\{\beta-(X'X)^{-1}X'X\beta -(X'X)^{-1}X'\epsilon\right\}\\=-\epsilon'X(X'X)^{-1}X'\epsilon$

Now

$\Sigma (Y_i - \hat Y_i)^2\\=\epsilon'X(X'X)^{-1}X'\epsilon-\epsilon'X(X'X)^{-1}X'\epsilon-\epsilon'X(X'X)^{-1}X'\epsilon+\epsilon'\epsilon\\=\epsilon'\epsilon-\epsilon'X(X'X)^{-1}X'\epsilon\\=\epsilon'\epsilon-\epsilon'P\epsilon$

$P$ is the projection matrix which is symmetric and idempotent

Now calculate the expectation.

$E[\Sigma (Y_i - \hat Y_i)^2]\\=E(\epsilon'\epsilon-\epsilon'P\epsilon)\\=E(\epsilon'\epsilon)-E(\epsilon'P\epsilon)\\=n\sigma^2-\sigma^2trace(P) \\\text{(Suppose tarace(P)}=k)$

$=(n-k)\sigma^2$

$\therefore \frac{\Sigma (Y_i - \hat Y_i)^2}{n-k}$ is the unbiased estimator of $\sigma^2$, $k$ is the number of parameters you want to estimate,such as you want to estimate $\beta_0$ for intercept and $\beta_1$ for one predictor, the $k$ will be equal to 2.

$\endgroup$
  • 1
    $\begingroup$ Question: why does $E(\epsilon'P\epsilon)=\sigma^2trace(P)$? I understand that $E(\epsilon' \epsilon) = \sigma^2$. But why did E(P) = trace(P)? $\endgroup$ – Adrian Jan 9 at 18:53
3
$\begingroup$

A less computationally intensive method would be $$ \begin{aligned} e&=y-\hat{y}\\ \Sigma(y[k]-\hat{y}[k])^2&=e^Te\\ y-\hat{y}&=\phi\theta-\phi\hat{\theta}+\epsilon\\ &=\phi\theta-\phi(\phi^T\phi)^{-1}\phi^Ty+\epsilon\\ &=\phi\theta-\phi(\phi^T\phi)^{-1}\phi^T(\phi\theta+\epsilon)+\epsilon\\ &=-\phi(\phi^T\phi)^{-1}\phi^T\epsilon+\epsilon\\ e&=(I-P)\epsilon\\ e^Te&=\epsilon^T(I-P^T-P+P^TP),\;\{P=P^T=P^TP\}\\ &=\epsilon^T\epsilon-\epsilon^TP\epsilon \end{aligned} $$ This is an easier method to get to the necessary step. You can then proceed further as explained by the previous answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.