Why is that? Why $TSS = ESS + RSS$ ,iff only we have an intercept(constant term) in our regression model?Why it doesn't work, when model doesn't include intercept?

  • $\begingroup$ See this post. stats.stackexchange.com/questions/233256/… $\endgroup$
    – Haitao Du
    Nov 28, 2016 at 16:23
  • $\begingroup$ @hxd1011 thank you for the comment. But it just provides the fact, that calculation of $R^2$ is different in two cases. I know that. I want to know technically why is it true. $\endgroup$ Nov 28, 2016 at 16:26

3 Answers 3


Here is my deduction, hope it can be helpful.

To start, let's break down the correlation between TSS, ESS, and RSS.

\begin{eqnarray*} TSS&=&\displaystyle\sum_{i}(y_{i}-\overline{y})^2\\ &=&\displaystyle\sum_{i}((y_{i} - \hat{y}_i)+(\hat{y}_i - \overline{y})^2\\ &=&\displaystyle\sum_{i}(y_{i} - \hat{y}_i)^2+\displaystyle\sum_{i}(\hat{y}_i - \overline{y})^2+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})\\ &=&ESS+RSS+2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y}) \end{eqnarray*}

We can see that there is a cross-term in the equation.

Given the fact that we are using linear regression model,

\begin{equation} \hat{y}_i = a + bx_i\\ \overline{y} = a + b\overline{x}\\ \hat{y}_i - \overline{y} = b(x_i - \overline{x})\\ y_{i} - \hat{y}_{i} = (y_{i} - \overline{y}_{i}) - (\hat{y}_{i} - \overline{y}_{i}) = (y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}) \end{equation}

Now, we apply above equations into the cross-term

\begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2b\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{x}_i - \overline{x})\\ &=&2b\displaystyle\sum_{i}((y_{i} - \overline{y}_{i}) - b(x_{i} - \overline{x}_{i}))(\hat{x}_i - \overline{x})\\ &=&2b(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - b\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2) \end{eqnarray*}

Finally, when the model has an intercept, we can get the best value of b by applying least square estimate \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}-\overline{x})(y_{i}-\overline{y})}{\displaystyle\sum_{i}(x_{i}-\overline{x})^2} \end{equation}

Therefore, \begin{eqnarray*} 2\displaystyle\sum_{i}(y_{i} - \hat{y}_i)(\hat{y}_i - \overline{y})&=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \hat{b}\displaystyle\sum_{i}(\hat{x}_i - \overline{x})^2)\\ &=&2\hat{b}(\displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}) - \displaystyle\sum_{i}(y_{i} - \overline{y}_{i})(\hat{x}_i - \overline{x}))\\ &=&0 \end{eqnarray*}

But, when the model does not have an intercept, the best value of b is \begin{equation} \hat{b} = \frac{\displaystyle\sum_{i}(x_{i}y_{i})}{\displaystyle\sum_{i}x_{i}^2} \end{equation}

In this case, we can not make the cross-term be zero.


To expand on @hxd1011's linked-to answer in the comments, \begin{align*} \text{TSS} &= \sum_i(y_i - \bar{y})^2 \\ &= \sum_{i}(y_i - \hat{y}_i + \hat{y}_i - \bar{y})^2\\ &= \sum_{i}(y_i - \hat{y}_i)^2 + \sum_i (\hat{y}_i - \bar{y})^2 + 2\sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) \\ &= \text{ESS} + \text{RSS} + 2\sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}). \end{align*} @hxd1011 is telling you that sometimes this cross term is $0$, and sometimes it is not.

For simplicity, let's say we only have one predictor. With an intercept, taking the derivatives of $\sum_i(y_i - \hat{y}_i)^2 = \sum_i(y_i - \beta_0 - \beta_1 x_i)^2$ with respect to both $\beta_0$ and $\beta_1$ implies that $\sum_i(y_i - \hat{y}_i) = 0$ and $\sum_i(y_i - \hat{y}_i)x_i =0$. These, taken together show that

\begin{align*} \sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) &= \sum_i(y_i - \hat{y}_i)\hat{y}_i - \bar{y} \sum_i(y_i - \hat{y}_i) \\ &= \hat{\beta}_0\sum_i(y_i - \hat{y}_i) + \hat{\beta}_1 \sum_i(y_i - \hat{y}_i)x_i- \bar{y} \sum_i(y_i - \hat{y}_i) \end{align*} is zero. So with an intercept, the cross term cancels out.

However, if you don't have an intercept, you take only one derivative (with respect to $\beta_1$, and setting this equal to $0$ you get $\sum_i(y_i - \beta_1 x_i)x_i = 0$. However, this alone does not tell you that the residuals sum to $0$, and it won't give you the cross term canceling out.

\begin{align*} \sum_i(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) &= \sum_i(y_i - \hat{y}_i)\hat{y}_i - \bar{y} \sum_i(y_i - \hat{y}_i) \\ &= \hat{\beta}_1 \sum_i(y_i - \hat{y}_i)x_i - \bar{y} \sum_i(y_i - \hat{y}_i) \\ &= - \bar{y} \sum_i(y_i - \hat{y}_i). \end{align*}


A bit of linear algebra may clarify matters somewhat. If you view $x=\{x_i\}$, $y=\{y_i\}$, and $\operatorname{1}=\{1\}$ (column of 1's) as column vectors in $\mathbb R^N$, then OLS regression with an intercept is obtained by simply projecting $y$ onto a vector $\hat{y}=\hat{\beta}_1x+\hat{\beta}_0\operatorname{1}\in\operatorname{span}(x,\operatorname{1})$. We have $$ESS = \Vert y-\hat{y}\Vert^2\text{,}$$ $$RSS = \Vert\hat{y}-\bar{y}\operatorname{1}\Vert^2\text{,}$$ and $$TSS= \Vert y-\bar{y}\operatorname{1}\Vert^2\text{.}$$ Now since $\hat{y}$ was projected from $y$ onto $\operatorname{span}(x,\operatorname{1})$ , $y-\hat{y}$ must be perpendicular to $\operatorname{span}(x,\operatorname{1})$, whereas $\hat{y}-\bar{y}\operatorname{1}$ lies in $\operatorname{span}(x,\operatorname{1})$. Therefore the vectors $y-\hat{y}$ and $\hat{y}-\bar{y}\operatorname{1}$ are orthogonal, and the result follows from the Pythagorean theorem.

Now consider the case where you regress without an intercept. In this case, you are projecting $y$ onto the one dimensional space $\operatorname{span}(x)$, which typically does not contain $\bar{y}\operatorname{1}$, so there is no reason to expect $y-\hat{y}$ to be perpendicular to $\hat{y}-\bar{y}\operatorname{1}$, hence the Pythagorean theorem does not apply.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.