1
$\begingroup$

Does RSS + ESS = TSS all the time?

I am thinking - does it hold if there is no intercept?

Does this apply for all regressions, or are there certain conditions that need to be met?

$\endgroup$
  • $\begingroup$ Dear @Mysterious, did you attempted to try it out yourself before asking? (Just asking) $\endgroup$ – ttnphns Dec 16 '11 at 11:22
  • $\begingroup$ you may find a useful answer here $\endgroup$ – probabilityislogic Dec 16 '11 at 13:12
1
$\begingroup$

I think it is the definition of TSS.

ESS is the explained sum of square, RSS is the residual sum of square.

ESS is the variation of the model.

RSS is defined as the variation we cannot explain by our model.

So obviously their sum is the total sum of square.

$\endgroup$
  • $\begingroup$ Or, ESS could be "error SS" and RSS could be "regression SS" $\endgroup$ – ttnphns Dec 16 '11 at 12:30
1
$\begingroup$

The equation holds true only when the model is linear regression and the model has an intercept(which means that the model must have a constant term).

Here is my deduction, hope it can be helpful.

To start, let's break down the correlation between TSS, ESS, and RSS.

\begin{eqnarray*} TSS&=&\displaystyle\sum_{i}(y_{i}-\overline{y})^2\\ &=&\displaystyle\sum_{i}((y_{i} - \hat{y}_i)+(\hat{y}_i - \overline{y})^2\\ &=&\displaystyle\sum_{i}(y_{i} - \hat{y}_i)^2+\displaystyle\sum_{i}(\hat{y}_i - \overline{y})^2+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})\\ &=&ESS+RSS+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y}) \end{eqnarray*}

We can see that there is a cross-term in the equation.

Given the fact that we are using linear regression model,

\begin{equation} \hat{y}_i = a + bx_i\\ \overline{y} = a + b\overline{x}\\ \hat{y}_i - \overline{y} = b(x_i - \overline{x})\\ y_{i} - \hat{y}_{i} = (y_{i} - \overline{y}_{i}) - (\hat{y}_{i} - \overline{y}_{i}) = (y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}) \end{equation}

Now, we apply above equations into the cross-term

\begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2b\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{x}_i - \overline{x})\\ &=&2b\displaystyle\sum_{i}((y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}))(\hat{x}_i - \overline{x})\\ &=&2b(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - b\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2) \end{eqnarray*}

Finally, when the model has an intercept, we can get the best value of b by applying least square estimate \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}-\overline{x})(y_{i}-\overline{y})}{\displaystyle\sum_{i}(x_{i}-\overline{x})^2} \end{equation}

Therefore, \begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \hat{b}\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2)\\ &=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}))\\ &=&0 \end{eqnarray*}

But, when the model does not have an intercept, the best value of b is \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}y_{i})}{\displaystyle\sum_{i}x_{i}^2} \end{equation}

In this case, we can not make the cross-term be zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.