Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Why are we using the squared residuals instead of the absolute residuals in OLS estimation?

My idea was that we use the square of the error values, so that residuals below the fitted line (which are then negative), would still have to be able to be added up to the positive errors. Otherwise, we could have an error of 0 simply because a huge positive error could cancel with a huge negative error.

So why do we square it, instead of just taking the absolute value? Is that because of the extra penalty for higher errors (instead of 2 being 2 times the error of 1, it is 4 times the error of 1 when we square it).

share|improve this question

marked as duplicate by Nick Stauner, gung, Peter Flom Apr 20 at 13:17

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

4  
OLS estimation basically minimises the sum of squared residuals. From a mathematical point of view, this requires taking the derivative. And there is less trouble in taking the derivative of $x \mapsto x^2$ than in taking the derivative of $x \mapsto |x|$. That's it. –  ocram Dec 16 '12 at 12:22
    
@ocram: "And there is less trouble in taking the derivative of x↦x2 than in taking the derivative of x↦|x|". I disagree! It's arguably easier to minimize |x-m| than it is to minimize (x-m)^2 (just find $m$ that equalizes $|\{x:x-m>0\}|$ and $|\{x:x-m<0\}|$). –  user603 Dec 16 '12 at 12:51
3  
Related - stats.stackexchange.com/questions/118/… –  Peter Ellis Dec 16 '12 at 23:43
5  
@user603 That's easy for finding a median, but for multiple regression it gets much harder. The OLS solution involves solving a linear system which is as easy as it gets. (Historically, the first efforts along these lines, c. 1755, indeed minimized the sum of residuals. There's a pretty geometric solution in the univariate model, but I don't think it generalizes to multiple explanatory variables.) –  whuber Dec 17 '12 at 2:20
3  
In The History of Statistics, Stephen Stigler recounts a 1755 analysis by Roger Boscovich, who argued first that the $L^1$ line passes through the centroid of the points, and then studied how the $L^1$ error varied as a line was rotated through the centroid. By sorting the slopes of the points (relative to the centroid, as an origin), he produced a $O(n\log(n))$ algorithm. See pp 46-49. –  whuber Dec 17 '12 at 15:28

4 Answers 4

up vote 12 down vote accepted

Both are done.

Least squares is easier, and the fact that for independent random variables "variances add" means that it's considerably more convenient; for examples, the ability to partition variances is particularly handy for comparing nested models. It's somewhat more efficient at the normal (least squares is maximum likelihood), which might seem to be a good justification -- however, some robust estimators with high breakdown can have surprisingly high efficiency at the normal.

But L1 norms are certainly used for regression problems and these days relatively often.

If you use R, you might find the discussion in section 5 here useful:

http://socserv.mcmaster.ca/jfox/Books/Companion/appendix/Appendix-Robust-Regression.pdf

(though the stuff before it on M estimation is also relevant, since it's also a special case of that)

share|improve this answer
    
This is a really nice reference. Thank you. –  PascalvKooten Dec 17 '12 at 8:55

I can't help quoting from Huber, Robust Statistics, p.10 on this (sorry the quote is too long to fit in a comment):

Two time-honored measures of scatter are the mean absolute deviation

$$d_n=\frac{1}{n}\sum|x_i-\bar{x}|$$

and the mean square deviation

$$s_n=\left[\frac{1}{n}\sum(x_i-\bar{x})^2\right]^{1/2}$$

There was a dispute between Eddington (1914, p.147) and Fisher (1920, footnote on p. 762) about the relative merits of $d_n$ and $s_n$.[...] Fisher seemingly settled the matter by pointing out that for normal observations $s_n$ is about 12% more efficient than $d_n$.

By the relation between the conditional mean $\hat{y}$ and the unconditional mean $\bar{x}$ a similar argument applies to the residuals.

share|improve this answer
    
I have to actually learn now about efficiency (and sufficiency, consistency etc), so it really got me curious to find out "12% more efficient" works... –  PascalvKooten Dec 16 '12 at 13:23
2  
Essentially, a more efficient estimator is one that needs fewer samples than a less efficient one to achieve a given accuracy. In this case, if we denote the sample size as $n$, it means that as $n\rightarrow\infty$, $d_n$ is $\sqrt{0.12}$ less accurate than $s_n$ –  user603 Dec 16 '12 at 13:24
    
Thanks, seems logical! I'm still curious though to how it can actually be more efficient... and why it is the case. Is there a way to proof that it is more efficient than the sum over all (xi-xbar)^4 then? –  PascalvKooten Dec 16 '12 at 13:28
4  
@Dualinity: the intuitive answer is that this because we define efficiency as the expectation of the squared distance between the estimated value $\bar{x}$ and the true value $\mu$ (e.g. $E(\bar{x}-\mu)^2$). By aligning the loss function used in the estimation of the parameter with that used to define efficiency, we maximize the efficiency of the estimated parameter. –  user603 Dec 16 '12 at 13:42

When the problem is expressed stochastically: $Y=aX+b+\epsilon$, where $\epsilon$ is normally distributed, the maximum likelihood estimate is the OLS estimate - not the minimum absolute deviation (MAD) estimate. So that's nice.

Furthermore, there is a strong link between OLS estimation and linear algebra. $\hat{Y}$ is a linear function of $Y$ --- in fact, it is a projection onto a subspace defined by the independent variables.

A lot of nice things happen with OLS --- MAD, not so much. And as @user603 points out, OLS are more efficient (where the normal model holds). It is less robust, of course.

share|improve this answer

One thing that has not been mentioned yet is uniqueness. The least squares approach always produces a single "best" answer. When minimizing the sum of the absolute value of the residuals it is possible that there may be an infinite number of lines that all have the same sum of absolute residuals (the minimum). Which of those line should be used?

share|improve this answer
    
Can you clarify the point you're trying to make, Greg? You seem to be making some statement regarding the coefficient estimates. But, certainly, in OLS the coefficients need not be unique. And, if there is more than one solution, we can immediately conclude there will be an infinite number of solutions. –  cardinal Dec 27 '12 at 1:34
    
@cardinal, I was just saying that when using least squares you get a single unique answer, with least absolute value you can get infinite which makes it harder to interpret the results. –  Greg Snow Dec 27 '12 at 1:57
    
Hi Greg, that is what I'm hoping you can clarify in your answer. You claim you get a single unique answer, but an answer to what? (Certainly not to the regression coefficients, necessarily.) –  cardinal Dec 27 '12 at 3:08
3  
I am refering to the least squares estimate of the regression coefficients being unique, though I was making the assumption (not originally stated) that the columns of the x matrix are linearly independent, or the x matrix is full rank, or ... (other ways of saying this): en.wikipedia.org/wiki/Linear_least_squares_(mathematics) –  Greg Snow Dec 27 '12 at 17:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.