5
$\begingroup$

One of the assumptions of linear regression is that the residual mean is zero. As far as I can tell though, the residual mean is always zero i.e. it is not an assumption, it is a fact.

The formula for calculating the least squares line means that the sum of all residuals is zero and the mean of all residuals is zero (even if the line is not actually linear).

Can someoen explain if/why this is not the case? Would appreciate an "english" response rather that jumping straight into the maths please!

$\endgroup$
4
  • 6
    $\begingroup$ The linear regression assumptions concern the model errors, not the model residuals. The model residuals estimate the (unknown) model errors. Since the model errors are assumed to have mean zero, then the model residuals "mimic" that and have an average of zero (hence a sum of zero). A good starting point for you would be to understand the difference between the model errors and the model residuals. $\endgroup$ Nov 18, 2018 at 23:54
  • $\begingroup$ See this link for more details on the difference between the model errors and model residuals: stats.stackexchange.com/questions/133389/…. $\endgroup$ Nov 19, 2018 at 15:29
  • 1
    $\begingroup$ Echoing @Isabella, it might be worthwhile pointing out that you use "residual mean" in two distinct ways. The residual mean assumption refers to the expectations of random variables in a model. The residual mean result--namely, "is always zero"--refers to a statistic (ie, a function of the data) generated by a certain kind of procedure (namely, ordinary least squares that includes a constant term). Clearing this semantic confusion ought to suffice to improve one's understanding. $\endgroup$
    – whuber
    Nov 19, 2018 at 15:37
  • $\begingroup$ youtube.com/watch?v=f6q4f3NiVNY I found this helpful in understanding this concept $\endgroup$
    – Alicia
    Aug 21, 2022 at 10:04

2 Answers 2

2
$\begingroup$

I think the above comments are quite clear. You have a true model first:

$$ y_i = x_i \beta + e_i $$

You assume the mean of the random variable, $e_i$, is zero. When you use OLS or some other methods to fit the model, you have already acknowledged this assumption. For example, OLS is minimizing the sum of square of residuals, which is kind of assuming that the error term tends to be small and, hence, the residual should be close to zero in some sense. Suppose the mean of residual is not zero and you don't have an intercept in this model, OLS might not make much sense as the errors might not be close to zero. Note you always think of a true model first, and then you discuss the estimation.

It's also worth mentioning that the mean of residual is not necessarily zero. It will be zero only when you have constant in the model.

$\endgroup$
2
$\begingroup$

The assumption underlying the linear model approach is not that the residual is mean-zero. That statement is tautological. The assumption is that the data, itself, obeys a linear distribution, $$ y = a x + b + \epsilon, $$ where $\epsilon$ is a mean-zero residual term and $b$ is a constant. The assumption is in the linear functional form of $y$, not in the fact that $\epsilon$ is mean-zero. The reason I call that latter statement "tautological", is because you can always redefine the component of the equation that's independent of $x$ as follows:

$$ b + \epsilon = c + \delta, $$ where $c$ is a constant, and $\delta$ is a residual with non-zero mean, but rather, a mean value equal to $b - c$.

Any random variable $\delta$ can be re-written as $(\delta - E[\delta]) + E[\delta]$. We know that $\delta -E[\delta]$ is a RV with zero mean by definition, and $E[\delta]$ is just a constant. In choosing the $x$-independent part to read as "$b + \epsilon$", where $\epsilon$ is mean-zero RV and $b$ is a constant, that's just another way of saying "the portion of $y$ that's independent of $x$ is a random variable, $\delta$, and we'll refer to $\delta - E[\delta]$ as "$\epsilon$", and $E[\delta]$ as "$b$".

It's true that a least-squares optimization is guaranteed to have zero-mean residual on the training set on which you did the optimization. However, that is independent of the assumption. When you say that your assumption is $y = a x + b + \epsilon$, that's not an assumption about your model, but rather, an assumption about the data itself. You're only training a linear model to learn what those parameters are.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.