6
$\begingroup$

From Wikipedia: https://en.wikipedia.org/wiki/Residual_sum_of_squares, the RSS is the average squared error between true value $y$, and the predicted value $\hat y$.

Then according to: https://en.wikipedia.org/wiki/Total_sum_of_squares, the TSS is the squared error between the true value $y$, and the average of all $y$.


However, I don't understand this line under the explanation for TSS:

[...] the total sum of squares equals the explained sum of squares plus the residual sum of squares.


If we plot RSS on the graph, it would look like:

enter image description here


TSS Plot:

enter image description here


ESS Plot:

enter image description here


According to the images, the residual (unexplained) value is actually larger than the TSS. Is there something I'm not following?

$\endgroup$
7
$\begingroup$

You have the total sum of squares being $\displaystyle \sum_i ({y}_i-\bar{y})^2$

which you can write as $\displaystyle \sum_i ({y}_i-\hat{y}_i+\hat{y}_i-\bar{y})^2 $

i.e. as $\displaystyle \sum_i ({y}_i-\hat{y}_i)^2+2\sum_i ({y}_i-\hat{y}_i)(\hat{y}_i-\bar{y}) +\sum_i(\hat{y}_i-\bar{y})^2$ where

  • the first summation term is the residual sum of squares,
  • the second is zero (if not then there is correlation, suggesting there are better values of $\hat{y}_i$) and
  • the third is the explained sum of squares

Since you have sums of squares, they must be non-negative and so the residual sum of squares must be less than the total sum of squares

$\endgroup$
  • $\begingroup$ Hi, thanks for your answer. It may be a bit too late, but can you elaborate on how the 2nd term means mathematically? Looking at the equation, I don't think I can come up with the conclusion that it needs to be 0. Thanks again! $\endgroup$ – user1157751 Mar 9 '17 at 7:50
  • $\begingroup$ Can I say this for the second term: For linear regression, the amount we over estimate and under estimate is zero, so when we sum $y_i$-$\hat y_i$, we will get 0? $\endgroup$ – user1157751 Mar 9 '17 at 18:29
  • $\begingroup$ Yes you should get $0$ since the average of each should be $\bar{y}$ and so they have the same sums. But my point is subtly different: the residuals should be uncorrelated with the predicted values as a direct result of the linear regression $\endgroup$ – Henry Mar 9 '17 at 18:35
  • $\begingroup$ Thanks for your reply again! I don't see a connection between "the residuals should be uncorrelated with the predicted values as a direct result of the linear regression" with any of the terms (1, 2, and 3). Can you give me some hints? Thanks $\endgroup$ – user1157751 Mar 9 '17 at 18:44
  • 1
    $\begingroup$ $\sum_i ({y}_i-\hat{y}_i)(\hat{y}_i-\bar{y})$ is the covariance of the residuals and the predicted values. Try an ordinary least squares regression and see that it is zero $\endgroup$ – Henry Mar 9 '17 at 21:15
2
$\begingroup$

This chart was very helpful for me. enter image description here

Via

$\endgroup$
  • $\begingroup$ could you please reference chart? $\endgroup$ – Carlos ST Aug 3 '18 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.