One definition of the Residual Sum of Squares is:

$$ S_r = (y-X\hat{\beta})^T(y-X\hat{\beta}) $$

And I think I understand it.

Now I have seen a different definition:

$$ S_r = y^Ty- \hat{\beta}^TX^TX\hat{\beta} $$

I think they supposed to be equal but I can't see why.

I can write (I leave out the \hat on $\beta$ to make the typing easier):

$$ \begin{aligned} S_r &= y^T(y-X\beta) - \beta^TX^T(y-X\beta)\\ &= y^Ty - y^TX\beta - \beta^TX^T y + \beta^TX^TX\beta \end{aligned} $$

and then to make both equations equal I would need to see that $\beta^TX^TX\beta = \beta^TX^T y$ which I don't.

How, do you show that the equations are equal?

  • 5
    $\begingroup$ Hint: $\hat{\beta} = (X'X)^{-1}X'y$. $\endgroup$ – Wolfgang May 18 '17 at 20:52
  • 2
    $\begingroup$ @hxd1011 You cannot just pick some arbitrary values for beta. Again, see my hint. $\endgroup$ – Wolfgang May 18 '17 at 21:01
  • 1
    $\begingroup$ So it is not linear algebra, but assume $\beta$ is optimal !! $\endgroup$ – Haitao Du May 18 '17 at 21:03
  • 2
    $\begingroup$ @hxd1011 This code exemplifies the situation. Run it: n <- 10; x <- seq(-2, 2, length.out=n); beta <- c(1,-2); y <- beta[1] + beta[2]*x + rnorm(n); fit <- lm(y ~ x); Sr.1 <- sum(resid(fit)^2); Sr.2 <- sum(y^2) - sum(predict(fit)^2); print(c(RSS.1=Sr.1, RSS.2=Sr.2)) This is just the Pythagorean Theorem: the first formula is the square of one leg of a right triangle while the second formula subtracts the square of the other leg from the square of the hypotenuse. $\endgroup$ – whuber May 18 '17 at 21:06
  • 1
    $\begingroup$ @hxd1011 That is not the reason, the reason is you are using the "true" $\beta$ and not the estimated $\hat{\beta}=(X^\top X)^{-1} X^\top y$ $\endgroup$ – Josh May 19 '17 at 4:03

In math, generally there are more than one way to prove a equation. I think the previous Answers are correct, but differ from your approach. Your approach is correct also, but you need one more step.

At first, you missed hat on $\beta$ in your last equation. Using the fact that $\hat \beta =(X^TX)^{-1}X^Ty$, we have

$\hat\beta^TX^Ty=y^TX(X^TX)^{-1}X^Ty = y^TX(X^TX)^{-1}(X^TX)(X^TX)^{-1}X^Ty = \hat\beta^TX^TX\hat\beta$

So finished your steps.


The most compact way to see the equality is to use the orthogonal projection matrix $P = X(X'X)^{-1}X'$ and the residual-maker matrix $M = I-P$. Both these matrices are symmetric, $P'=P,\;\; M'=M$ and idempotent $PP=P,\;\; MM = M$. We have

$$X\hat{\beta} = Py, \;\;y-X\hat{\beta}= My$$

Then $$S_r = (y-X\hat{\beta})'(y-X\hat{\beta}) = (My)'(My) = y'M'My = y'My$$

$$ = y'(I-P)y = y'y - y'Py = y'y-(Py)'Py=y'y- \hat{\beta}'X'X\hat{\beta}$$


As @whuber said in the comment, it is essentially Pythagorean Theorem.

Where Let error


and prediction

$$ p=X\hat\beta $$

The first equation is $$ e^Te=\|e\|^2 $$ and second equation is $y^Ty-p^Tp=\|y\|^2-\|p\|^2$. Because vector $e$ and vector $p$ are perpendicular, we have $\|y\|^2-\|p\|^2=\|e\|^2$.

Details can also be found in Gilbert Strangs's linear algebra book. Sample chapter here

enter image description here

  • 1
    $\begingroup$ The definition of $p$ in your answer seems to miss the $\text{^}$ in $\hat{\beta}$. The distinction between $\beta$ and $\hat{\beta}$ is crucial to correctly disentangling the OP's misunderstanding. $\endgroup$ – Glen_b -Reinstate Monica May 18 '17 at 23:57
  • $\begingroup$ You have introduced a new problem above the previous one; the term "error" would apply to the term as you originally had it, to $y-X\beta$. If you really do mean $y-X\hat{\beta}$ (as it now says), that's normally referred to as residual, not error. Errors are unobserved, residuals are in effect an estimate of the errors. Again, the distinction is important (and I think related the original misunderstanding in the question). $\endgroup$ – Glen_b -Reinstate Monica May 19 '17 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.