0
$\begingroup$

Assuming a simple linear regression framework, $y_i=\beta_{0}+\beta_{1}.x_{i} + \epsilon_{i}$, why can't I estimate $\beta_{0}$ and $\beta_{1}$ by optimizing the sum of residuals such that the sum is equal to zero ?

(i.e. not setting $\sum_{i=1}^{N}\hat{\epsilon_{i}} = 0$)

Let me know if the question is awkwardly asked :-)

$\endgroup$
4
  • $\begingroup$ Linear algebra informs us that the solution will not be unique: there will be an entire line of solutions. Which solution will you pick?? $\endgroup$
    – whuber
    Nov 11 at 19:48
  • $\begingroup$ What says you can’t do that? $$\hat\beta_{marlou} \in\bigg\{(a,b)\in\mathbb R^2\bigg\vert \sum_{i=1}^N \big(y_i -(a+bx_i)\big)=0\bigg\}$$ Now pick something from that set, and call that your estimate of the regression coefficients. $\endgroup$
    – Dave
    Nov 11 at 19:49
  • $\begingroup$ Thank You @Dave for replying ! :)Intuitively I have the impression (and maybe my intuition is wrong) that finding a line such that the mean deviation from the line is zero is a good idea ! I thought that there was only 1 solution to this set. But in principle I still don't 100% understand why we choose to minimise the square of the error rather than the error directly. $\endgroup$ Nov 11 at 23:28
  • 2
    $\begingroup$ Here’s some intuition. Let’s say I have to predict two numbers, say the final score of a basketball game (each team has a score). I predict the New York Knicks to score 108 and the Brooklyn Nets to score 95. They play the game, and Brooklyn wins 109 to 94. Add up my errors; they are zero. But have I made good predictions about the basketball game? $\endgroup$
    – Dave
    Nov 11 at 23:36
1
$\begingroup$

Why can't I estimate $\beta_{0}$ and $\beta_{1}$ by optimizing the sum of residuals such that the sum is equal to zero ?

You can find some $\beta_0$ and $\beta_1$ such that $\sum_{i=1}^n \epsilon_i =0$ but the solution won't be unique and it is also not meaningfull.

The image below shows an example with two points.

example

By tilting the line you can increase the errors, but you can do this in a way such that there are negative and positive errors that cancel each other.

For any $\beta_1$ you can compute...

If:

$$\sum_{i=1}^n \epsilon_i = \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i) = \sum_{i=1}^n y_i - n \beta_0 - \sum_{i=1}^n \beta_1 x_i = 0$$

Then:

$$\beta_0 = \frac{1}{n} \sum_{i=1}^n (y_i-\beta_1 x_i)$$


Minimizing the sum of $\epsilon_i$ makes no sense, but minimizing the sum of the absolute residuals $\sum_{i=1}^n \vert \epsilon_i \vert$ is something that is done a lot and is called quantile regression.

$\endgroup$
0
$\begingroup$

You cannot directly observe $\epsilon_i$. Statisticians make an assumption that these errors are iid $N(0, 1)$. So "optimizing" to reduce that to zero doesn't quite make sense to me.

Finding an estimator such that $\sum\limits^n_{i=1} (y_i - \beta_0 + \beta_1 x_i) =0$ may not have a unique solution. As whuber pointed out, minimizing this also does not work since you can over predict everything and make the residuals arbitrarily negative.

So why can't you set $\sum\limits^n_{i=1} |\hat \epsilon_i|=0$? Well that solution (a perfect fit) may not exist.

That's why the least squares estimate minimizes $\sum\limits^n_{i=1} (y_i - \beta_0 + \beta_1 x_i)^2$. Which will have a unique solution for simple linear regression (except for the weird edge case where x_i's are constant).

$\endgroup$
6
  • $\begingroup$ You lost me at "may not exist." The OLS solution always exists and it is always the case that the sum of its residuals is zero. I don't follow your graphical example at all, because the sum of those residuals looks pretty close to zero to me. Are you perhaps thinking of setting the sum of absolute residuals to zero? $\endgroup$
    – whuber
    Nov 11 at 21:42
  • $\begingroup$ Ah, you're right. Fixed. $\endgroup$
    – decaf
    Nov 11 at 22:46
  • $\begingroup$ Hello @shotime ! Thanks for replying ! :) I realize that I should have asked my question in a better looking way: I don't understand why (in general in a linear regression model) we are trying to find 𝛽0 and 𝛽1 by minimizing the sum of squared error and not the sum of error (I wasn't thinking about the sum of the absolute value, I even thought the OLS method was preferable for the convexity of the function). I wonder because intuitively I have the impression (and maybe my intuition is wrong) that finding a line such that the mean deviation from the line is zero is a good idea. $\endgroup$ Nov 11 at 23:10
  • $\begingroup$ By the way thank you @whuber also ! From the response of you and Dave above and Shotime's, I think that I understand that 1) there is not only one (B0,B1) couples such that the sum of errors is equal to zero and 2) we are therefore finding B0 and B1 by minimizing the sum of square error because it allows us to find the minimum of the sum of absolute value (or squared) which is more relevant because positive and negative residuals can cancel each other which makes not sense Am I right ? Am I right ? $\endgroup$ Nov 11 at 23:20
  • $\begingroup$ The edited version of this answer still doesn't work. First, you cannot minimize the sum of the residuals, because that sum can be made arbitrarily negative. Reinterpreting the question as setting the sum of absolute residuals to zero trivializes it because, as you point out, that happens only with perfect fits. Finally, the OLS solution is not necessarily unique. $\endgroup$
    – whuber
    Nov 12 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.