Algebraic definition of a residual from a regression

Can anyone give some advice on how to start proving this algebraically?

Define the residual from a regression (one independent variable) algebraically and show that:

1. the mean of the residuals is zero
2. the correlation of the residuals and the independent variable is zero
-
I presume this is homework? What have you tried so far? –  csgillespie Mar 1 '11 at 13:50
can you give a more explicit statement of the model you're considering? I ask because, in general, the first thing you're asked to prove is false! –  cardinal Mar 1 '11 at 13:50

Suppose you have the following regression model:

$$y_i=\alpha+\beta x_i+\varepsilon_i$$

Least squares problem looks for $\alpha$ and $\beta$ which minimize the following function:

$$g(\alpha,\beta)=\sum_{i=1}^n(y_i-\alpha-\beta x_i)^2$$

Solution for this problem will satisfy

$$\frac{\partial g}{\partial \alpha}=0, \quad \frac{\partial g}{\partial \beta}=0.$$

Try differencing and look hard at the resulting expressions. You will see that this answers your problem.

Note that @cardinal is right, if you do not include $\alpha$, your first statement is false.

Update This is might be considered non-algebraic solution, so please state more clearly what do you mean by algebraic. If this is not helpful, I will retract my answer, which really is a long comment.

-
Well I am supposed to have an algebraic proof that the mean of the residuals is zero. I assumed that by showing that the sum of the residuals is zero, then one could state that the mean is similarly zero, but I am not sure if that is accurate. I feel that something is missing.. –  user3487 Mar 1 '11 at 19:21
@cassetete, did you try to differentiate $g$? You are right mean will be zero when sum will be zero. –  mpiktas Mar 1 '11 at 19:33
It finally became clear to me. I just worked it out, many thanks to all! –  user3487 Mar 1 '11 at 19:51
@cassetete, congratulations! It would be good if you posted your solution as an answer. So that somebody can find it by googling it. Or you can accept my answer, wink, wink, hint, hint, if it was the proverbial push over the cliff. –  mpiktas Mar 1 '11 at 20:01