Why do residuals in linear regression always sum to zero when an intercept is included?

I'm taking a course on regression models and one of the properties provided for linear regression is that the residuals always sum to zero when an intercept is included.

Can someone provide a good explanation for why this is the case?

• You might like to first ponder the closely related but simpler question of why in a univariate sample, the residuals you obtain by subtracting the sample mean from each value also sum to 0. (Try following the algebra through if you can.) – Glen_b Jan 7 '16 at 1:50
• As soon as you recognize that "sum to zero" means "orthogonal to one of the explanatory variables" the answer becomes geometrically obvious. – whuber May 5 '17 at 19:57

This follows directly from the normal equations, i.e. the equations that the OLS estimator solves,

$$\mathbf{X}^{\prime} \underbrace{\left( \mathbf{y} - \mathbf{X} \mathbf{b} \right)}_{\mathbf{e}} = 0$$

The vector inside the parentheses is of course the residual vector or the projection of $\mathbf{y}$ onto the orthogonal complement of the column space of $X$, if you like linear algebra. Now including a vector of ones in the $\mathbf{X}$ matrix, which by the way doesn't have to be in the first column as is conventionally done, leads to

$$\mathbf{1}^{\prime} \mathbf{e} = 0 \implies \sum_{i=1}^n e_i = 0$$

In the two-variable problem this is even simpler to see, as minimizing the sum of squared residuals brings us to

$$\sum_{i=1}^n \left(y_i - a - b x_i \right) = 0$$

when we take the derivative with respect to the intercept. From this then we proceed to obtain the familiar estimator

$$a = \bar{y} - b \bar{x}$$

where again we see that the construction of our estimators imposes this condition.

In case you are looking for a rather intuitiv explanation.

In some sense the linear regression model is nothing but a fancy mean. To find the arithmetic mean $\bar{x}$ over some values $x_1, x_2, \dots, x_n$, we find a value that is a measure of centrality in a sense that the sum of all deviations (where each deviation is defined as $u_i = x_i - \bar{x}$) to the right of the mean value are equal to the sum of all the deviations to the left of that mean. There is no inherent reason why this measure is a good, let alone the best way to describe the mean of a sample, but it is certainly intuitiv and practial. The important point is, that by defining the arithmetic mean in this way, it necessarily follows that once we constructed the arithmetic mean, all deviations from that mean must sum to zero by definition!

In a linear regression, this is no different. We fit the line such that the sum of all differences between our fitted values (which are on the regression line) and the actual values that are above the line is exactly equal to the sum of all differences between the regression line and all values below the line. Again, there is no inherent reason, why this is the best way to construct a fit, but it is straightforward and intuitively appealing. Just as with the arithmetic mean: by constructing our fitted values in this way, it necessarily follows, by construction, that all deviations from that line must sum to zero for otherwise this just wouldnt be an OLS regession.

• +1 for straightforward, simple and intuitive answer! – user218970 Oct 7 '18 at 9:40

When an intercept is included in multiple linear regression, $$\hat{y}_i = \beta_0 + \beta_1x_{i,1} + \beta_2x_{i,2} +…+ \beta_px_{i,p}$$ In Least squares regression, the sum of the squares of the errors is minimized. $$SSE=\displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= \sum_{i=1}^n\left(y_i - \hat{y_i} \right)^2= \sum_{i=1}^n\left(y_i -\beta_0- \beta_1x_{i,1}-\beta_2x_{i,2}-…- \beta_px_{i,p} \right)^2$$ Take the partial derivative of SSE with respect to $$\beta_0$$ and setting it to zero. $$\frac{\partial{SSE}}{\partial{\beta_0}} = \sum_{i=1}^n 2\left(y_i -\beta_0- \beta_1x_{i,1}-\beta_2x_{i,2}-…- \beta_px_{i,p} \right)^1 (-1) =-2\displaystyle\sum\limits_{i=1}^ne_i=0$$ Hence, the residuals always sum to zero when an intercept is included in linear regression.

A simple derivation using matrix algebra:

$\sum e$ can be written as $1^Te$

Then

$1^Te = 1^T(M_x y)$ where $M_x$ is the orthogonal matrix. Since $M_x$ is symmetric we can rearrange so that $(M_x1)^Ty$

which equals zero if $M_x$ and $1$ are orthogonal, which is the case if the matrix of the regressors $x$ contains the intercept (a vector of $1$, indeed).

• I don't think this is right. – Michael Chernick May 5 '17 at 18:40
• If you explain why then I will be happy to learn something – Mino May 5 '17 at 20:16

A key observation is that because the model has intercept, $$1$$, which is the first column of design matrix $$X$$, can be written as $$1 = Xe,$$ where $$e$$ is a column vector with all zeros but the first component one. Also note, in matrix notation, the sum of residuals is just $$1^T(y - \hat{y})$$.

Therefore, \begin{align} & 1^T(y - \hat{y}) = 1^T(I - H)y \\ = & e^TX^T(I - X(X^TX)^{-1}X^T)y \\ = & e^T(X^T - X^TX(X^TX)^{-1}X^T)y \\ = & e^T(X^T - X^T)y \\ = & 0. \end{align}

• Could you explain the reasons for the steps in your derivation? The last line is especially mysterious: how does $0$ suddenly appear? – whuber Oct 9 at 19:37
• @whuber Because $AA^{-1} = I$. – Zhanxiong Oct 9 at 19:58
• Thank you: I'm sure your readers will appreciate the extra clarifying line. – whuber Oct 9 at 21:39