Linear regression - assume residual mean is zero

Question

One of the assumptions of linear regression is that the residual mean is zero. As far as I can tell though, the residual mean is always zero i.e. it is not an assumption, it is a fact.

The formula for calculating the least squares line means that the sum of all residuals is zero and the mean of all residuals is zero (even if the line is not actually linear).

Can someoen explain if/why this is not the case? Would appreciate an "english" response rather that jumping straight into the maths please!

Answer 1

I think the above comments are quite clear. You have a true model first:

$$ y_i = x_i \beta + e_i $$

You assume the mean of the random variable, $e_i$, is zero. When you use OLS or some other methods to fit the model, you have already acknowledged this assumption. For example, OLS is minimizing the sum of square of residuals, which is kind of assuming that the error term tends to be small and, hence, the residual should be close to zero in some sense. Suppose the mean of residual is not zero and you don't have an intercept in this model, OLS might not make much sense as the errors might not be close to zero. Note you always think of a true model first, and then you discuss the estimation.

It's also worth mentioning that the mean of residual is not necessarily zero. It will be zero only when you have constant in the model.

Answer 2

The assumption underlying the linear model approach is not that the residual is mean-zero. That statement is tautological. The assumption is that the data, itself, obeys a linear distribution, $$ y = a x + b + \epsilon, $$ where $\epsilon$ is a mean-zero residual term and $b$ is a constant. The assumption is in the linear functional form of $y$, not in the fact that $\epsilon$ is mean-zero. The reason I call that latter statement "tautological", is because you can always redefine the component of the equation that's independent of $x$ as follows:

$$ b + \epsilon = c + \delta, $$ where $c$ is a constant, and $\delta$ is a residual with non-zero mean, but rather, a mean value equal to $b - c$.

Any random variable $\delta$ can be re-written as $(\delta - E[\delta]) + E[\delta]$. We know that $\delta -E[\delta]$ is a RV with zero mean by definition, and $E[\delta]$ is just a constant. In choosing the $x$-independent part to read as "$b + \epsilon$", where $\epsilon$ is mean-zero RV and $b$ is a constant, that's just another way of saying "the portion of $y$ that's independent of $x$ is a random variable, $\delta$, and we'll refer to $\delta - E[\delta]$ as "$\epsilon$", and $E[\delta]$ as "$b$".

It's true that a least-squares optimization is guaranteed to have zero-mean residual on the training set on which you did the optimization. However, that is independent of the assumption. When you say that your assumption is $y = a x + b + \epsilon$, that's not an assumption about your model, but rather, an assumption about the data itself. You're only training a linear model to learn what those parameters are.