1
$\begingroup$

One of the properties of OLS is $\sum {\rm residuals} = 0$.

I understand the proof but my intuition doesn't understand why this is true. That is, why does minimising the sum of squares of residuals imply that the sum of the residuals = 0?

Do you have a simple explanation which doesn't involve a lot of computation?

$\endgroup$
  • 1
    $\begingroup$ Exactly which proof do you know? For instance, one proof I know (and teach) is that the residuals are orthogonal to the regressors (this is called the "normal equations"). Provided a constant is in the vector space spanned by the regressors, the residuals will be orthogonal to it, QED. In effect, there is nothing to be shown. This proof also (immediately) shows that it's not always the case that the residuals sum to zero! BTW, this relationship is unrelated to the usual meaning of "unbiased" in a regression context. $\endgroup$ – whuber Oct 28 '15 at 12:13
  • $\begingroup$ @whuber When you say that the residuals might not always sum to zero, are you referring to Regression Through the Origin, or perhaps you have a different situation in mind? $\endgroup$ – JohnK Oct 28 '15 at 13:50
  • 1
    $\begingroup$ @JohnK It includes regression through the origin as a prominent example, but can also include other situations. Note that even when a constant (aka "intercept") is not included in the model, one might be implicitly present anyway. This can happen especially in ANOVA-like situations, depending on how the variables are coded. Thus it would be inaccurate to characterize the cases where residuals sum to zero in terms of whether an intercept is explicitly present. $\endgroup$ – whuber Oct 28 '15 at 14:35
5
$\begingroup$

The constant term (the intercept) absorbs the non-zero mean of the error term, and permits us to assume safely that the "remaining" error-term has a zero mean. This why it is almost always wrong to not include a intercept, because then $E(u) \neq 0$, unless by chance.

More formally, since the the first order partial is:

$$ \frac{\partial}{\partial \beta_0} = -2 \sum ^n _{i=1} (y_i - \hat \beta_0 - \hat \beta_1 x_1) =0 $$

But the term in the sum is also the residuals, such that:

$$ \sum ^n _{i=1} (y_i - \hat \beta_0 - \hat \beta_1 x_1) = \sum^n _{i=1} \hat u_i= 0 $$

EDIT: Also note that this fact follows immediately from including the intercept. Unbiasedness, consistency, homoskedlasticity, normality or whatever else one might be willing to assume does not play any role.

$\endgroup$
  • $\begingroup$ Thanks, that what I was missing, thanks ! ps : what do you mean by "You can the worlds most biased estimator - OLS will still give the above." ? $\endgroup$ – psql Oct 28 '15 at 17:47
  • $\begingroup$ If you include the intercept, then the residuals will always (really always) sum to 0. Regardless of whether the model is unbiased or "good", or the worst (horrible) model one can imagine $\endgroup$ – Repmat Oct 28 '15 at 17:52
  • $\begingroup$ Is it ok to say Unbiaised => sum(residual) = 0 ? $\endgroup$ – psql Oct 28 '15 at 17:59
  • $\begingroup$ You are missing the point, unbiasedness does not play a role. If you include an intercept, then the sum of residuals will be 0 (no matter what else you might assume) $\endgroup$ – Repmat Oct 28 '15 at 18:08
  • $\begingroup$ I wanted to say if a estimator is unbiased, does it implies the sum of its residual = 0. I'm talking about implication not causation. $\endgroup$ – psql Oct 28 '15 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.