41
$\begingroup$

Why are we using the squared residuals instead of the absolute residuals in OLS estimation?

My idea was that we use the square of the error values, so that residuals below the fitted line (which are then negative), would still have to be able to be added up to the positive errors. Otherwise, we could have an error of 0 simply because a huge positive error could cancel with a huge negative error.

So why do we square it, instead of just taking the absolute value? Is that because of the extra penalty for higher errors (instead of 2 being 2 times the error of 1, it is 4 times the error of 1 when we square it).

Improve this question
$\endgroup$
7
  • 9
    $\begingroup$ OLS estimation basically minimises the sum of squared residuals. From a mathematical point of view, this requires taking the derivative. And there is less trouble in taking the derivative of $x \mapsto x^2$ than in taking the derivative of $x \mapsto |x|$. That's it. $\endgroup$
    – ocram
    Commented Dec 16, 2012 at 12:22
  • 1
    $\begingroup$ @ocram: "And there is less trouble in taking the derivative of x↦x2 than in taking the derivative of x↦|x|". I disagree! It's arguably easier to minimize |x-m| than it is to minimize (x-m)^2 (just find $m$ that equalizes $|\{x:x-m>0\}|$ and $|\{x:x-m<0\}|$). $\endgroup$
    – user603
    Commented Dec 16, 2012 at 12:51
  • 3
    $\begingroup$ Related - stats.stackexchange.com/questions/118/… $\endgroup$
    – Peter Ellis
    Commented Dec 16, 2012 at 23:43
  • 9
    $\begingroup$ @user603 That's easy for finding a median, but for multiple regression it gets much harder. The OLS solution involves solving a linear system which is as easy as it gets. (Historically, the first efforts along these lines, c. 1755, indeed minimized the sum of residuals. There's a pretty geometric solution in the univariate model, but I don't think it generalizes to multiple explanatory variables.) $\endgroup$
    – whuber ♦
    Commented Dec 17, 2012 at 2:20
  • 5
    $\begingroup$ In The History of Statistics, Stephen Stigler recounts a 1755 analysis by Roger Boscovich, who argued first that the $L^1$ line passes through the centroid of the points, and then studied how the $L^1$ error varied as a line was rotated through the centroid. By sorting the slopes of the points (relative to the centroid, as an origin), he produced a $O(n\log(n))$ algorithm. See pp 46-49. $\endgroup$
    – whuber ♦
    Commented Dec 17, 2012 at 15:28

4 Answers 4

Reset to default
24
$\begingroup$

Both are done.

Least squares is easier, and the fact that for independent random variables "variances add" means that it's considerably more convenient; for examples, the ability to partition variances is particularly handy for comparing nested models. It's somewhat more efficient at the normal (least squares is maximum likelihood), which might seem to be a good justification -- however, some robust estimators with high breakdown can have surprisingly high efficiency at the normal.

But L1 norms are certainly used for regression problems and these days relatively often.

If you use R, you might find the discussion in section 5 here useful:

https://socialsciences.mcmaster.ca/jfox/Books/Companion/appendices/Appendix-Robust-Regression.pdf

(though the stuff before it on M estimation is also relevant, since it's also a special case of that)

Improve this answer
$\endgroup$
0
27
$\begingroup$

I can't help quoting from Huber, Robust Statistics, p.10 on this (sorry the quote is too long to fit in a comment):

Two time-honored measures of scatter are the mean absolute deviation

$$d_n=\frac{1}{n}\sum|x_i-\bar{x}|$$

and the mean square deviation

$$s_n=\left[\frac{1}{n}\sum(x_i-\bar{x})^2\right]^{1/2}$$

There was a dispute between Eddington (1914, p.147) and Fisher (1920, footnote on p. 762) about the relative merits of $d_n$ and $s_n$.[...] Fisher seemingly settled the matter by pointing out that for normal observations $s_n$ is about 12% more efficient than $d_n$.

By the relation between the conditional mean $\hat{y}$ and the unconditional mean $\bar{x}$ a similar argument applies to the residuals.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ I have to actually learn now about efficiency (and sufficiency, consistency etc), so it really got me curious to find out "12% more efficient" works... $\endgroup$
    – PascalVKooten
    Commented Dec 16, 2012 at 13:23
  • 5
    $\begingroup$ Essentially, a more efficient estimator is one that needs fewer samples than a less efficient one to achieve a given accuracy. In this case, if we denote the sample size as $n$, it means that as $n\rightarrow\infty$, $d_n$ is $\sqrt{0.12}$ less accurate than $s_n$ $\endgroup$
    – user603
    Commented Dec 16, 2012 at 13:24
  • 7
    $\begingroup$ @Dualinity: the intuitive answer is that this because we define efficiency as the expectation of the squared distance between the estimated value $\bar{x}$ and the true value $\mu$ (e.g. $E(\bar{x}-\mu)^2$). By aligning the loss function used in the estimation of the parameter with that used to define efficiency, we maximize the efficiency of the estimated parameter. $\endgroup$
    – user603
    Commented Dec 16, 2012 at 13:42
  • 2
    $\begingroup$ Eddington's approach was empirical and based in real-life data. Fisher's approach was mathematical and based in idealized conditions: normal distributions, error-free measurements. Fisher proof doesn't apply when errors are present. 1% error is enough to invert roles and make $d_n$ more efficient than $s_n$ $\endgroup$
    – juanrga
    Commented Aug 6, 2017 at 10:46
  • 3
    $\begingroup$ @juanrga Yes, and Tukey mentions that Fisher was the only statistician he surveyed in the mid 50's that had any sense of how fragile ('1% of deviating data sampled from a re-scaled normal') this optimality result was. $\endgroup$
    – user603
    Commented Aug 6, 2017 at 11:59
18
$\begingroup$

One thing that has not been mentioned yet is uniqueness. The least squares approach always produces a single "best" answer if the matrix of explanatory variables is full rank. When minimizing the sum of the absolute value of the residuals it is possible that there may be an infinite number of lines that all have the same sum of absolute residuals (the minimum). Which of those line should be used?

Improve this answer
$\endgroup$
13
  • 1
    $\begingroup$ Can you clarify the point you're trying to make, Greg? You seem to be making some statement regarding the coefficient estimates. But, certainly, in OLS the coefficients need not be unique. And, if there is more than one solution, we can immediately conclude there will be an infinite number of solutions. $\endgroup$
    – cardinal
    Commented Dec 27, 2012 at 1:34
  • $\begingroup$ @cardinal, I was just saying that when using least squares you get a single unique answer, with least absolute value you can get infinite which makes it harder to interpret the results. $\endgroup$
    – Greg Snow
    Commented Dec 27, 2012 at 1:57
  • 3
    $\begingroup$ I am refering to the least squares estimate of the regression coefficients being unique, though I was making the assumption (not originally stated) that the columns of the x matrix are linearly independent, or the x matrix is full rank, or ... (other ways of saying this): en.wikipedia.org/wiki/Linear_least_squares_(mathematics) $\endgroup$
    – Greg Snow
    Commented Dec 27, 2012 at 17:01
  • 1
    $\begingroup$ The non-uniqueness could be sorted out with an additional criterion, like requiring $\hat{\beta}$ of minimum norm. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Aug 8, 2015 at 10:10
  • 1
    $\begingroup$ That is very true. In case of Linear Regression, it never fits a single best fitted line if mod of errors is used as compared to square of error. But it still doesn't clarify that why not power 4 or 8. Why 2?? $\endgroup$
    – Mayukh Sarkar
    Commented Nov 10, 2017 at 14:19
17
$\begingroup$

When the problem is expressed stochastically: $Y=aX+b+\epsilon$, where $\epsilon$ is normally distributed, the maximum likelihood estimate is the OLS estimate - not the minimum absolute deviation (MAD) estimate. So that's nice.

Furthermore, there is a strong link between OLS estimation and linear algebra. $\hat{Y}$ is a linear function of $Y$ --- in fact, it is a projection onto a subspace defined by the independent variables.

A lot of nice things happen with OLS --- MAD, not so much. And as @user603 points out, OLS are more efficient (where the normal model holds). It is less robust, of course.

Improve this answer
$\endgroup$
1

Not the answer you're looking for? Browse other questions tagged or ask your own question.