I wanted to do simple calculations for sum of squares for linear regression using a very simple example.

1) Let's assume that we have 3 obervations [x,y]: [[1,1], [2,2], [3,3]].

2) I created simple regression model which for each x is assigning 3. So it is a flat line.

3) For that regression line I wanted to calculate: total sum of suares (TSS), residual sum of squares (RSS) and explained sum of squares (ESS).

enter image description here enter image description here enter image description here

As far as I know, TSS = RSS + ESS and here we have:

2 = 5 + 3

I can't figure out what I did wrong or TSS = RSS + ESS is aplicable only under certain conditions?


TSS = RSS + ESS only applies when the sum of the residuals (not the sum of the squared residuals, mind) is 0 (or equivalently, when the residuals have mean 0). In your example, all residuals are positive, and therefore their sum is also positive, and not 0, so the equality does not hold here.

Note that the sum and mean of the residuals are guaranteed to be 0 when the regression model includes an intercept, and has been fit to the data using ordinary least squares. This is typical in practice, and explains the relevance of the TSS = RSS + ESS formula.

(In your example, you could make the residuals have mean 0 by changing your model to predict a constant value of 2 rather than 3. If you retry your calculations with that model, you'll see the equality does hold in that case.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.