3
$\begingroup$

One definition of the Residual Sum of Squares is:

$$ S_r = (y-X\hat{\beta})^T(y-X\hat{\beta}) $$

And I think I understand it.

Now I have seen a different definition:

$$ S_r = y^Ty- \hat{\beta}^TX^TX\hat{\beta} $$

I think they supposed to be equal but I can't see why.

I can write (I leave out the \hat on $\beta$ to make the typing easier):

$$ \begin{aligned} S_r &= y^T(y-X\beta) - \beta^TX^T(y-X\beta)\\ &= y^Ty - y^TX\beta - \beta^TX^T y + \beta^TX^TX\beta \end{aligned} $$

and then to make both equations equal I would need to see that $\beta^TX^TX\beta = \beta^TX^T y$ which I don't.

How, do you show that the equations are equal?

Improve this question
$\endgroup$
9
  • 5
    $\begingroup$ Hint: $\hat{\beta} = (X'X)^{-1}X'y$. $\endgroup$
    – Wolfgang
    Commented May 18, 2017 at 20:52
  • 2
    $\begingroup$ @hxd1011 You cannot just pick some arbitrary values for beta. Again, see my hint. $\endgroup$
    – Wolfgang
    Commented May 18, 2017 at 21:01
  • 1
    $\begingroup$ So it is not linear algebra, but assume $\beta$ is optimal !! $\endgroup$
    – Haitao Du
    Commented May 18, 2017 at 21:03
  • 2
    $\begingroup$ @hxd1011 This code exemplifies the situation. Run it: n <- 10; x <- seq(-2, 2, length.out=n); beta <- c(1,-2); y <- beta[1] + beta[2]*x + rnorm(n); fit <- lm(y ~ x); Sr.1 <- sum(resid(fit)^2); Sr.2 <- sum(y^2) - sum(predict(fit)^2); print(c(RSS.1=Sr.1, RSS.2=Sr.2)) This is just the Pythagorean Theorem: the first formula is the square of one leg of a right triangle while the second formula subtracts the square of the other leg from the square of the hypotenuse. $\endgroup$
    – whuber ♦
    Commented May 18, 2017 at 21:06
  • 1
    $\begingroup$ @hxd1011 That is not the reason, the reason is you are using the "true" $\beta$ and not the estimated $\hat{\beta}=(X^\top X)^{-1} X^\top y$ $\endgroup$
    – Josh
    Commented May 19, 2017 at 4:03

3 Answers 3

Reset to default
2
$\begingroup$

In math, generally there are more than one way to prove a equation. I think the previous Answers are correct, but differ from your approach. Your approach is correct also, but you need one more step.

At first, you missed hat on $\beta$ in your last equation. Using the fact that $\hat \beta =(X^TX)^{-1}X^Ty$, we have

$\hat\beta^TX^Ty=y^TX(X^TX)^{-1}X^Ty = y^TX(X^TX)^{-1}(X^TX)(X^TX)^{-1}X^Ty = \hat\beta^TX^TX\hat\beta$

So finished your steps.

Improve this answer
$\endgroup$
3
$\begingroup$

The most compact way to see the equality is to use the orthogonal projection matrix $P = X(X'X)^{-1}X'$ and the residual-maker matrix $M = I-P$. Both these matrices are symmetric, $P'=P,\;\; M'=M$ and idempotent $PP=P,\;\; MM = M$. We have

$$X\hat{\beta} = Py, \;\;y-X\hat{\beta}= My$$

Then $$S_r = (y-X\hat{\beta})'(y-X\hat{\beta}) = (My)'(My) = y'M'My = y'My$$

$$ = y'(I-P)y = y'y - y'Py = y'y-(Py)'Py=y'y- \hat{\beta}'X'X\hat{\beta}$$

Improve this answer
$\endgroup$
0
$\begingroup$

As @whuber said in the comment, it is essentially Pythagorean Theorem.

Where Let error

$$e=y-X\hat\beta$$

and prediction

$$ p=X\hat\beta $$

The first equation is $$ e^Te=\|e\|^2 $$ and second equation is $y^Ty-p^Tp=\|y\|^2-\|p\|^2$. Because vector $e$ and vector $p$ are perpendicular, we have $\|y\|^2-\|p\|^2=\|e\|^2$.

Details can also be found in Gilbert Strangs's linear algebra book. Sample chapter here

enter image description here

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The definition of $p$ in your answer seems to miss the $\text{^}$ in $\hat{\beta}$. The distinction between $\beta$ and $\hat{\beta}$ is crucial to correctly disentangling the OP's misunderstanding. $\endgroup$
    – Glen_b
    Commented May 18, 2017 at 23:57
  • $\begingroup$ You have introduced a new problem above the previous one; the term "error" would apply to the term as you originally had it, to $y-X\beta$. If you really do mean $y-X\hat{\beta}$ (as it now says), that's normally referred to as residual, not error. Errors are unobserved, residuals are in effect an estimate of the errors. Again, the distinction is important (and I think related the original misunderstanding in the question). $\endgroup$
    – Glen_b
    Commented May 19, 2017 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.