Why are we using the squared residuals instead of the absolute residuals in OLS estimation?

My idea was that we use the square of the error values, so that residuals below the fitted line (which are then negative), would still have to be able to be added up to the positive errors. Otherwise, we could have an error of 0 simply because a huge positive error could cancel with a huge negative error.

So why do we square it, instead of just taking the absolute value? Is that because of the extra penalty for higher errors (instead of 2 being 2 times the error of 1, it is 4 times the error of 1 when we square it).

  • 8
    $\begingroup$ OLS estimation basically minimises the sum of squared residuals. From a mathematical point of view, this requires taking the derivative. And there is less trouble in taking the derivative of $x \mapsto x^2$ than in taking the derivative of $x \mapsto |x|$. That's it. $\endgroup$ – ocram Dec 16 '12 at 12:22
  • 1
    $\begingroup$ @ocram: "And there is less trouble in taking the derivative of x↦x2 than in taking the derivative of x↦|x|". I disagree! It's arguably easier to minimize |x-m| than it is to minimize (x-m)^2 (just find $m$ that equalizes $|\{x:x-m>0\}|$ and $|\{x:x-m<0\}|$). $\endgroup$ – user603 Dec 16 '12 at 12:51
  • 3
    $\begingroup$ Related - stats.stackexchange.com/questions/118/… $\endgroup$ – Peter Ellis Dec 16 '12 at 23:43
  • 7
    $\begingroup$ @user603 That's easy for finding a median, but for multiple regression it gets much harder. The OLS solution involves solving a linear system which is as easy as it gets. (Historically, the first efforts along these lines, c. 1755, indeed minimized the sum of residuals. There's a pretty geometric solution in the univariate model, but I don't think it generalizes to multiple explanatory variables.) $\endgroup$ – whuber Dec 17 '12 at 2:20
  • 4
    $\begingroup$ In The History of Statistics, Stephen Stigler recounts a 1755 analysis by Roger Boscovich, who argued first that the $L^1$ line passes through the centroid of the points, and then studied how the $L^1$ error varied as a line was rotated through the centroid. By sorting the slopes of the points (relative to the centroid, as an origin), he produced a $O(n\log(n))$ algorithm. See pp 46-49. $\endgroup$ – whuber Dec 17 '12 at 15:28

Both are done.

Least squares is easier, and the fact that for independent random variables "variances add" means that it's considerably more convenient; for examples, the ability to partition variances is particularly handy for comparing nested models. It's somewhat more efficient at the normal (least squares is maximum likelihood), which might seem to be a good justification -- however, some robust estimators with high breakdown can have surprisingly high efficiency at the normal.

But L1 norms are certainly used for regression problems and these days relatively often.

If you use R, you might find the discussion in section 5 here useful:


(though the stuff before it on M estimation is also relevant, since it's also a special case of that)


I can't help quoting from Huber, Robust Statistics, p.10 on this (sorry the quote is too long to fit in a comment):

Two time-honored measures of scatter are the mean absolute deviation


and the mean square deviation


There was a dispute between Eddington (1914, p.147) and Fisher (1920, footnote on p. 762) about the relative merits of $d_n$ and $s_n$.[...] Fisher seemingly settled the matter by pointing out that for normal observations $s_n$ is about 12% more efficient than $d_n$.

By the relation between the conditional mean $\hat{y}$ and the unconditional mean $\bar{x}$ a similar argument applies to the residuals.

  • $\begingroup$ I have to actually learn now about efficiency (and sufficiency, consistency etc), so it really got me curious to find out "12% more efficient" works... $\endgroup$ – PascalVKooten Dec 16 '12 at 13:23
  • 5
    $\begingroup$ Essentially, a more efficient estimator is one that needs fewer samples than a less efficient one to achieve a given accuracy. In this case, if we denote the sample size as $n$, it means that as $n\rightarrow\infty$, $d_n$ is $\sqrt{0.12}$ less accurate than $s_n$ $\endgroup$ – user603 Dec 16 '12 at 13:24
  • 7
    $\begingroup$ @Dualinity: the intuitive answer is that this because we define efficiency as the expectation of the squared distance between the estimated value $\bar{x}$ and the true value $\mu$ (e.g. $E(\bar{x}-\mu)^2$). By aligning the loss function used in the estimation of the parameter with that used to define efficiency, we maximize the efficiency of the estimated parameter. $\endgroup$ – user603 Dec 16 '12 at 13:42
  • 1
    $\begingroup$ Eddington's approach was empirical and based in real-life data. Fisher's approach was mathematical and based in idealized conditions: normal distributions, error-free measurements. Fisher proof doesn't apply when errors are present. 1% error is enough to invert roles and make $d_n$ more efficient than $s_n$ $\endgroup$ – juanrga Aug 6 '17 at 10:46
  • 3
    $\begingroup$ @juanrga Yes, and Tukey mentions that Fisher was the only statistician he surveyed in the mid 50's that had any sense of how fragile ('1% of deviating data sampled from a re-scaled normal') this optimality result was. $\endgroup$ – user603 Aug 6 '17 at 11:59

One thing that has not been mentioned yet is uniqueness. The least squares approach always produces a single "best" answer if the matrix of explanatory variables is full rank. When minimizing the sum of the absolute value of the residuals it is possible that there may be an infinite number of lines that all have the same sum of absolute residuals (the minimum). Which of those line should be used?

  • $\begingroup$ Can you clarify the point you're trying to make, Greg? You seem to be making some statement regarding the coefficient estimates. But, certainly, in OLS the coefficients need not be unique. And, if there is more than one solution, we can immediately conclude there will be an infinite number of solutions. $\endgroup$ – cardinal Dec 27 '12 at 1:34
  • $\begingroup$ @cardinal, I was just saying that when using least squares you get a single unique answer, with least absolute value you can get infinite which makes it harder to interpret the results. $\endgroup$ – Greg Snow Dec 27 '12 at 1:57
  • $\begingroup$ Hi Greg, that is what I'm hoping you can clarify in your answer. You claim you get a single unique answer, but an answer to what? (Certainly not to the regression coefficients, necessarily.) $\endgroup$ – cardinal Dec 27 '12 at 3:08
  • 3
    $\begingroup$ I am refering to the least squares estimate of the regression coefficients being unique, though I was making the assumption (not originally stated) that the columns of the x matrix are linearly independent, or the x matrix is full rank, or ... (other ways of saying this): en.wikipedia.org/wiki/Linear_least_squares_(mathematics) $\endgroup$ – Greg Snow Dec 27 '12 at 17:01
  • $\begingroup$ The non-uniqueness could be sorted out with an additional criterion, like requiring $\hat{\beta}$ of minimum norm. $\endgroup$ – kjetil b halvorsen Aug 8 '15 at 10:10

When the problem is expressed stochastically: $Y=aX+b+\epsilon$, where $\epsilon$ is normally distributed, the maximum likelihood estimate is the OLS estimate - not the minimum absolute deviation (MAD) estimate. So that's nice.

Furthermore, there is a strong link between OLS estimation and linear algebra. $\hat{Y}$ is a linear function of $Y$ --- in fact, it is a projection onto a subspace defined by the independent variables.

A lot of nice things happen with OLS --- MAD, not so much. And as @user603 points out, OLS are more efficient (where the normal model holds). It is less robust, of course.


Not the answer you're looking for? Browse other questions tagged or ask your own question.