# Residual sum of squares, two different equations, why are they equal?

One definition of the Residual Sum of Squares is:

$$S_r = (y-X\hat{\beta})^T(y-X\hat{\beta})$$

And I think I understand it.

Now I have seen a different definition:

$$S_r = y^Ty- \hat{\beta}^TX^TX\hat{\beta}$$

I think they supposed to be equal but I can't see why.

I can write (I leave out the \hat on $\beta$ to make the typing easier):

\begin{aligned} S_r &= y^T(y-X\beta) - \beta^TX^T(y-X\beta)\\ &= y^Ty - y^TX\beta - \beta^TX^T y + \beta^TX^TX\beta \end{aligned}

and then to make both equations equal I would need to see that $\beta^TX^TX\beta = \beta^TX^T y$ which I don't.

How, do you show that the equations are equal?

• Hint: $\hat{\beta} = (X'X)^{-1}X'y$. – Wolfgang May 18 '17 at 20:52
• @hxd1011 You cannot just pick some arbitrary values for beta. Again, see my hint. – Wolfgang May 18 '17 at 21:01
• So it is not linear algebra, but assume $\beta$ is optimal !! – Haitao Du May 18 '17 at 21:03
• @hxd1011 This code exemplifies the situation. Run it: n <- 10; x <- seq(-2, 2, length.out=n); beta <- c(1,-2); y <- beta[1] + beta[2]*x + rnorm(n); fit <- lm(y ~ x); Sr.1 <- sum(resid(fit)^2); Sr.2 <- sum(y^2) - sum(predict(fit)^2); print(c(RSS.1=Sr.1, RSS.2=Sr.2)) This is just the Pythagorean Theorem: the first formula is the square of one leg of a right triangle while the second formula subtracts the square of the other leg from the square of the hypotenuse. – whuber May 18 '17 at 21:06
• @hxd1011 That is not the reason, the reason is you are using the "true" $\beta$ and not the estimated $\hat{\beta}=(X^\top X)^{-1} X^\top y$ – Josh May 19 '17 at 4:03

In math, generally there are more than one way to prove a equation. I think the previous Answers are correct, but differ from your approach. Your approach is correct also, but you need one more step.

At first, you missed hat on $\beta$ in your last equation. Using the fact that $\hat \beta =(X^TX)^{-1}X^Ty$, we have

$\hat\beta^TX^Ty=y^TX(X^TX)^{-1}X^Ty = y^TX(X^TX)^{-1}(X^TX)(X^TX)^{-1}X^Ty = \hat\beta^TX^TX\hat\beta$

The most compact way to see the equality is to use the orthogonal projection matrix $P = X(X'X)^{-1}X'$ and the residual-maker matrix $M = I-P$. Both these matrices are symmetric, $P'=P,\;\; M'=M$ and idempotent $PP=P,\;\; MM = M$. We have

$$X\hat{\beta} = Py, \;\;y-X\hat{\beta}= My$$

Then $$S_r = (y-X\hat{\beta})'(y-X\hat{\beta}) = (My)'(My) = y'M'My = y'My$$

$$= y'(I-P)y = y'y - y'Py = y'y-(Py)'Py=y'y- \hat{\beta}'X'X\hat{\beta}$$

As @whuber said in the comment, it is essentially Pythagorean Theorem.

Where Let error

$$e=y-X\hat\beta$$

and prediction

$$p=X\hat\beta$$

The first equation is $$e^Te=\|e\|^2$$ and second equation is $y^Ty-p^Tp=\|y\|^2-\|p\|^2$. Because vector $e$ and vector $p$ are perpendicular, we have $\|y\|^2-\|p\|^2=\|e\|^2$.

Details can also be found in Gilbert Strangs's linear algebra book. Sample chapter here

• The definition of $p$ in your answer seems to miss the $\text{^}$ in $\hat{\beta}$. The distinction between $\beta$ and $\hat{\beta}$ is crucial to correctly disentangling the OP's misunderstanding. – Glen_b -Reinstate Monica May 18 '17 at 23:57
• You have introduced a new problem above the previous one; the term "error" would apply to the term as you originally had it, to $y-X\beta$. If you really do mean $y-X\hat{\beta}$ (as it now says), that's normally referred to as residual, not error. Errors are unobserved, residuals are in effect an estimate of the errors. Again, the distinction is important (and I think related the original misunderstanding in the question). – Glen_b -Reinstate Monica May 19 '17 at 22:29