Confused with Residual Sum of Squares and Total Sum of Squares

Question

From Wikipedia: https://en.wikipedia.org/wiki/Residual_sum_of_squares, the RSS is the average squared error between true value $y$, and the predicted value $\hat y$.

Then according to: https://en.wikipedia.org/wiki/Total_sum_of_squares, the TSS is the squared error between the true value $y$, and the average of all $y$.

However, I don't understand this line under the explanation for TSS:

[...] the total sum of squares equals the explained sum of squares plus the residual sum of squares.

If we plot RSS on the graph, it would look like:

TSS Plot:

ESS Plot:

According to the images, the residual (unexplained) value is actually larger than the TSS. Is there something I'm not following?

That's why the summation is important. As to why you can partition it see stats.stackexchange.com/questions/258284/… — Łukasz Grad, Mar 6, 2017 at 23:20
@ŁukaszGrad, I looked at the post and unfortunately I don't really get it. Can you elaborate a bit further? Thanks — user1157751, Mar 6, 2017 at 23:24
This may also help you: stats.stackexchange.com/questions/256726/… and this perhaps too: stats.stackexchange.com/questions/255973/… And maybe this one: stats.stackexchange.com/questions/256344/… — Stefan, Mar 6, 2017 at 23:25
About "the residual is larger than the TSS": the case in your graph is not the typical one. Most of the points will be closer to the predicted line than to the average. — matiasg, Aug 30 at 10:41

Henry · Accepted Answer · 2017-03-07 00:05:02Z

13

You have the total sum of squares being $\displaystyle \sum_i ({y}_i-\bar{y})^2$

which you can write as $\displaystyle \sum_i ({y}_i-\hat{y}_i+\hat{y}_i-\bar{y})^2 $

i.e. as $\displaystyle \sum_i ({y}_i-\hat{y}_i)^2+2\sum_i ({y}_i-\hat{y}_i)(\hat{y}_i-\bar{y}) +\sum_i(\hat{y}_i-\bar{y})^2$ where

the first summation term is the residual sum of squares,
the second is zero (if not then there is correlation, suggesting there are better values of $\hat{y}_i$) and
the third is the explained sum of squares

Since you have sums of squares, they must be non-negative and so the residual sum of squares must be less than the total sum of squares

answered Mar 7, 2017 at 0:05

Henry

37.2k1 gold badge76 silver badges125 bronze badges

$\begingroup$ Hi, thanks for your answer. It may be a bit too late, but can you elaborate on how the 2nd term means mathematically? Looking at the equation, I don't think I can come up with the conclusion that it needs to be 0. Thanks again! $\endgroup$
– user1157751
Mar 9, 2017 at 7:50
$\begingroup$ Can I say this for the second term: For linear regression, the amount we over estimate and under estimate is zero, so when we sum $y_i$-$\hat y_i$, we will get 0? $\endgroup$
– user1157751
Mar 9, 2017 at 18:29
$\begingroup$ Yes you should get $0$ since the average of each should be $\bar{y}$ and so they have the same sums. But my point is subtly different: the residuals should be uncorrelated with the predicted values as a direct result of the linear regression $\endgroup$
– Henry
Mar 9, 2017 at 18:35
$\begingroup$ Thanks for your reply again! I don't see a connection between "the residuals should be uncorrelated with the predicted values as a direct result of the linear regression" with any of the terms (1, 2, and 3). Can you give me some hints? Thanks $\endgroup$
– user1157751
Mar 9, 2017 at 18:44
2

$\begingroup$ $\sum_i ({y}_i-\hat{y}_i)(\hat{y}_i-\bar{y})$ is the covariance of the residuals and the predicted values. Try an ordinary least squares regression and see that it is zero $\endgroup$
– Henry
Mar 9, 2017 at 21:15

| Show 1 more comment

Estimate the estimators · Accepted Answer · 2018-08-04 16:22:32Z

6

This chart was very helpful for me.

answered Jun 20, 2018 at 12:24

5519 silver badges18 bronze badges

1

$\begingroup$ could you please reference chart? $\endgroup$
– Carlos S Traynor
Aug 3, 2018 at 11:39
1

$\begingroup$ Excellent explanation! Thank you! $\endgroup$
– Beginner
Nov 19, 2021 at 16:33

Add a comment |

2 Answers 2