Total sum of squares(TSS) is not equal ESS +RSS, when the model doesn't include intercept of ones

Question

Why is that? Why $TSS = ESS + RSS$ ,iff only we have an intercept(constant term) in our regression model?Why it doesn't work, when model doesn't include intercept?

Answer 1

Here is my deduction, hope it can be helpful.

To start, let's break down the correlation between TSS, ESS, and RSS.

\begin{eqnarray*} TSS&=&\displaystyle\sum_{i}(y_{i}-\overline{y})^2\\ &=&\displaystyle\sum_{i}((y_{i} - \hat{y}_i)+(\hat{y}_i - \overline{y})^2\\ &=&\displaystyle\sum_{i}(y_{i} - \hat{y}_i)^2+\displaystyle\sum_{i}(\hat{y}_i - \overline{y})^2+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})\\ &=&ESS+RSS+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y}) \end{eqnarray*}

We can see that there is a cross-term in the equation.

Given the fact that we are using linear regression model,

\begin{equation} \hat{y}_i = a + bx_i\\ \overline{y} = a + b\overline{x}\\ \hat{y}_i - \overline{y} = b(x_i - \overline{x})\\ y_{i} - \hat{y}_{i} = (y_{i} - \overline{y}_{i}) - (\hat{y}_{i} - \overline{y}_{i}) = (y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}) \end{equation}

Now, we apply above equations into the cross-term

\begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2b\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{x}_i - \overline{x})\\ &=&2b\displaystyle\sum_{i}((y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}))(\hat{x}_i - \overline{x})\\ &=&2b(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - b\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2) \end{eqnarray*}

Finally, when the model has an intercept, we can get the best value of b by applying least square estimate \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}-\overline{x})(y_{i}-\overline{y})}{\displaystyle\sum_{i}(x_{i}-\overline{x})^2} \end{equation}

Therefore, \begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \hat{b}\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2)\\ &=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}))\\ &=&0 \end{eqnarray*}

But, when the model does not have an intercept, the best value of b is \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}y_{i})}{\displaystyle\sum_{i}x_{i}^2} \end{equation}

In this case, we can not make the cross-term be zero.

Answer 2

To expand on @hxd1011's linked-to answer in the comments, \begin{align*} \text{TSS} &= \sum_i(y_i - \bar{y})^2 \\ &= \sum_{i}(y_i - \hat{y}_i + \hat{y}_i - \bar{y})^2\\ &= \sum_{i}(y_i - \hat{y}_i)^2 + \sum_i (\hat{y}_i - \bar{y})^2 + 2\sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) \\ &= \text{ESS} + \text{RSS} + 2\sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}). \end{align*} @hxd1011 is telling you that sometimes this cross term is $0$, and sometimes it is not.

For simplicity, let's say we only have one predictor. With an intercept, taking the derivatives of $\sum_i(y_i - \hat{y}_i)^2 = \sum_i(y_i - \beta_0 - \beta_1 x_i)^2$ with respect to both $\beta_0$ and $\beta_1$ implies that $\sum_i(y_i - \hat{y}_i) = 0$ and $\sum_i(y_i - \hat{y}_i)x_i =0$. These, taken together show that

\begin{align*} \sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) &= \sum_i(y_i - \hat{y}_i)\hat{y}_i - \bar{y} \sum_i(y_i - \hat{y}_i) \\ &= \hat{\beta}_0\sum_i(y_i - \hat{y}_i) + \hat{\beta}_1 \sum_i(y_i - \hat{y}_i)x_i- \bar{y} \sum_i(y_i - \hat{y}_i) \end{align*} is zero. So with an intercept, the cross term cancels out.

However, if you don't have an intercept, you take only one derivative (with respect to $\beta_1$, and setting this equal to $0$ you get $\sum_i(y_i - \beta_1 x_i)x_i = 0$. However, this alone does not tell you that the residuals sum to $0$, and it won't give you the cross term canceling out.

\begin{align*} \sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) &= \sum_i(y_i - \hat{y}_i)\hat{y}_i - \bar{y} \sum_i(y_i - \hat{y}_i) \\ &= \hat{\beta}_1 \sum_i(y_i - \hat{y}_i)x_i - \bar{y} \sum_i(y_i - \hat{y}_i) \\ &= - \bar{y} \sum_i(y_i - \hat{y}_i). \end{align*}

Answer 3

A bit of linear algebra may clarify matters somewhat. If you view $x=\{x_i\}$, $y=\{y_i\}$, and $\operatorname{1}=\{1\}$ (column of 1's) as column vectors in $\mathbb R^N$, then OLS regression with an intercept is obtained by simply projecting $y$ onto a vector $\hat{y}=\hat{\beta}_1x+\hat{\beta}_0\operatorname{1}\in\operatorname{span}(x,\operatorname{1})$. We have $$ESS = \Vert y-\hat{y}\Vert^2\text{,}$$ $$RSS = \Vert\hat{y}-\bar{y}\operatorname{1}\Vert^2\text{,}$$ and $$TSS= \Vert y-\bar{y}\operatorname{1}\Vert^2\text{.}$$ Now since $\hat{y}$ was projected from $y$ onto $\operatorname{span}(x,\operatorname{1})$ , $y-\hat{y}$ must be perpendicular to $\operatorname{span}(x,\operatorname{1})$, whereas $\hat{y}-\bar{y}\operatorname{1}$ lies in $\operatorname{span}(x,\operatorname{1})$. Therefore the vectors $y-\hat{y}$ and $\hat{y}-\bar{y}\operatorname{1}$ are orthogonal, and the result follows from the Pythagorean theorem.

Now consider the case where you regress without an intercept. In this case, you are projecting $y$ onto the one dimensional space $\operatorname{span}(x)$, which typically does not contain $\bar{y}\operatorname{1}$, so there is no reason to expect $y-\hat{y}$ to be perpendicular to $\hat{y}-\bar{y}\operatorname{1}$, hence the Pythagorean theorem does not apply.