Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm taking a course on regression models and one of the properties provided for linear regression is that the residuals always sum to zero when an intercept is included.

Can someone provide a good explanation for why this is the case?

share|improve this question
2  
You might like to first ponder the closely related but simpler question of why in a univariate sample, the residuals you obtain by subtracting the sample mean from each value also sum to 0. (Try following the algebra through if you can.) – Glen_b Jan 7 at 1:50

This follows directly from the normal equations, i.e. the equations that the OLS estimator solves,

$$\mathbf{X}^{\prime} \underbrace{\left( \mathbf{y} - \mathbf{X} \mathbf{b} \right)}_{\mathbf{e}} = 0 $$

The vector inside the parentheses is of course the residual vector or the projection of $\mathbf{y}$ onto the orthogonal complement of the column space of $X$, if you like linear algebra. Now including a vector of ones in the $\mathbf{X}$ matrix, which by the way doesn't have to be in the first column as is conventionally done, leads to

$$\mathbf{1}^{\prime} \mathbf{e} = 0 \implies \sum_{i=1}^n e_i = 0$$

In the two-variable problem this is even simpler to see, as minimizing the sum of squared residuals brings us to

$$\sum_{i=1}^n \left(y_i - a - b x_i \right) = 0$$

when we take the derivative with respect to the intercept. From this then we proceed to obtain the familiar estimator

$$a = \bar{y} - b \bar{x}$$

where again we see that the construction of our estimators imposes this condition.

share|improve this answer

In case you are looking for a rather intuitiv explanation.

In some sense the linear regression model is nothing but a fancy mean. To find the arithmetic mean $\bar{x}$ over some values $x_1, x_2, \dots, x_n$, we find a value that is a measure of centrality in a sense that the sum of all deviations (where each deviation is defined as $u_i = x_i - \bar{x}$) to the right of the mean value are equal to the sum of all the deviations to the left of that mean. There is no inherent reason why this measure is a good, let alone the best way to describe the mean of a sample, but it is certainly intuitiv and practial. The important point is, that by defining the arithmetic mean in this way, it necessarily follows that once we constructed the arithmetic mean, all deviations from that mean must sum to zero by definition!

In a linear regression, this is no different. We fit the line such that the sum of all differences between our fitted values (which are on the regression line) and the actual values that are above the line is exactly equal to the sum of all differences between the regression line and all values below the line. Again, there is no inherent reason, why this is the best way to construct a fit, but it is straightforward and intuitively appealing. Just as with the arithmetic mean: by constructing our fitted values in this way, it necessarily follows, by construction, that all deviations from that line must sum to zero for otherwise this just wouldnt be an OLS regession.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.