A question about the sum of ESS and RSS

Question

Does RSS + ESS = TSS all the time?

I am thinking - does it hold if there is no intercept?

Does this apply for all regressions, or are there certain conditions that need to be met?

Answer 1

I think it is the definition of TSS.

ESS is the explained sum of square, RSS is the residual sum of square.

ESS is the variation of the model.

RSS is defined as the variation we cannot explain by our model.

So obviously their sum is the total sum of square.

Answer 2

The equation holds true only when the model is linear regression and the model has an intercept(which means that the model must have a constant term).

Here is my deduction, hope it can be helpful.

To start, let's break down the correlation between TSS, ESS, and RSS.

\begin{eqnarray*} TSS&=&\displaystyle\sum_{i}(y_{i}-\overline{y})^2\\ &=&\displaystyle\sum_{i}((y_{i} - \hat{y}_i)+(\hat{y}_i - \overline{y})^2\\ &=&\displaystyle\sum_{i}(y_{i} - \hat{y}_i)^2+\displaystyle\sum_{i}(\hat{y}_i - \overline{y})^2+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})\\ &=&ESS+RSS+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y}) \end{eqnarray*}

We can see that there is a cross-term in the equation.

Given the fact that we are using linear regression model,

\begin{equation} \hat{y}_i = a + bx_i\\ \overline{y} = a + b\overline{x}\\ \hat{y}_i - \overline{y} = b(x_i - \overline{x})\\ y_{i} - \hat{y}_{i} = (y_{i} - \overline{y}_{i}) - (\hat{y}_{i} - \overline{y}_{i}) = (y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}) \end{equation}

Now, we apply above equations into the cross-term

\begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2b\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{x}_i - \overline{x})\\ &=&2b\displaystyle\sum_{i}((y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}))(\hat{x}_i - \overline{x})\\ &=&2b(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - b\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2) \end{eqnarray*}

Finally, when the model has an intercept, we can get the best value of b by applying least square estimate \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}-\overline{x})(y_{i}-\overline{y})}{\displaystyle\sum_{i}(x_{i}-\overline{x})^2} \end{equation}

Therefore, \begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \hat{b}\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2)\\ &=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}))\\ &=&0 \end{eqnarray*}

But, when the model does not have an intercept, the best value of b is \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}y_{i})}{\displaystyle\sum_{i}x_{i}^2} \end{equation}

In this case, we can not make the cross-term be zero.