11
$\begingroup$

From Wikipedia: https://en.wikipedia.org/wiki/Residual_sum_of_squares, the RSS is the average squared error between true value $y$, and the predicted value $\hat y$.

Then according to: https://en.wikipedia.org/wiki/Total_sum_of_squares, the TSS is the squared error between the true value $y$, and the average of all $y$.


However, I don't understand this line under the explanation for TSS:

[...] the total sum of squares equals the explained sum of squares plus the residual sum of squares.


If we plot RSS on the graph, it would look like:

enter image description here


TSS Plot:

enter image description here


ESS Plot:

enter image description here


According to the images, the residual (unexplained) value is actually larger than the TSS. Is there something I'm not following?

$\endgroup$
4

2 Answers 2

13
$\begingroup$

You have the total sum of squares being $\displaystyle \sum_i ({y}_i-\bar{y})^2$

which you can write as $\displaystyle \sum_i ({y}_i-\hat{y}_i+\hat{y}_i-\bar{y})^2 $

i.e. as $\displaystyle \sum_i ({y}_i-\hat{y}_i)^2+2\sum_i ({y}_i-\hat{y}_i)(\hat{y}_i-\bar{y}) +\sum_i(\hat{y}_i-\bar{y})^2$ where

  • the first summation term is the residual sum of squares,
  • the second is zero (if not then there is correlation, suggesting there are better values of $\hat{y}_i$) and
  • the third is the explained sum of squares

Since you have sums of squares, they must be non-negative and so the residual sum of squares must be less than the total sum of squares

$\endgroup$
6
  • $\begingroup$ Hi, thanks for your answer. It may be a bit too late, but can you elaborate on how the 2nd term means mathematically? Looking at the equation, I don't think I can come up with the conclusion that it needs to be 0. Thanks again! $\endgroup$ Mar 9, 2017 at 7:50
  • $\begingroup$ Can I say this for the second term: For linear regression, the amount we over estimate and under estimate is zero, so when we sum $y_i$-$\hat y_i$, we will get 0? $\endgroup$ Mar 9, 2017 at 18:29
  • $\begingroup$ Yes you should get $0$ since the average of each should be $\bar{y}$ and so they have the same sums. But my point is subtly different: the residuals should be uncorrelated with the predicted values as a direct result of the linear regression $\endgroup$
    – Henry
    Mar 9, 2017 at 18:35
  • $\begingroup$ Thanks for your reply again! I don't see a connection between "the residuals should be uncorrelated with the predicted values as a direct result of the linear regression" with any of the terms (1, 2, and 3). Can you give me some hints? Thanks $\endgroup$ Mar 9, 2017 at 18:44
  • 2
    $\begingroup$ $\sum_i ({y}_i-\hat{y}_i)(\hat{y}_i-\bar{y})$ is the covariance of the residuals and the predicted values. Try an ordinary least squares regression and see that it is zero $\endgroup$
    – Henry
    Mar 9, 2017 at 21:15
6
$\begingroup$

This chart was very helpful for me. enter image description here

Via

$\endgroup$
2
  • 1
    $\begingroup$ could you please reference chart? $\endgroup$ Aug 3, 2018 at 11:39
  • 1
    $\begingroup$ Excellent explanation! Thank you! $\endgroup$
    – Beginner
    Nov 19, 2021 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.