56
$\begingroup$

When we conduct linear regression $y=ax+b$ to fit a bunch of data points $(x_1,y_1),(x_2,y_2),...,(x_n,y_n)$, the classic approach minimizes the squared error. I have long been puzzled by a question that will minimizing the squared error yield the same result as minimizing the absolute error? If not, why minimizing squared error is better? Is there any reason other than "the objective function is differentiable"?

Squared error is also widely used to evaluate model performance, but absolute error is less popular. Why squared error is more commonly used than the absolute error? If taking derivatives is not involved, calculating absolute error is as easy as calculating squared error, then why squared error is so prevalent? Is there any unique advantage that can explain its prevalence?

Thank you.

Improve this question
$\endgroup$
2
  • $\begingroup$ There is always some optimization problem behind and you want be able to compute gradients to find minimum/maximum. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Apr 18, 2015 at 2:21
  • 14
    $\begingroup$ $x^2 < |x|$ for $x \in (-1,1)$ and $x^2 > |x|$ if $|x| > 1$. Thus, squared error penalizes large errors more than does absolute error and is more forgiving of small errors than absolute error is. This accords well with what many think is an appropriate way of doing things. $\endgroup$
    – Dilip Sarwate
    Commented Apr 18, 2015 at 3:19

5 Answers 5

Reset to default
71
$\begingroup$

Minimizing square errors (MSE) is definitely not the same as minimizing absolute deviations (MAD) of errors. MSE provides the mean response of $y$ conditioned on $x$, while MAD provides the median response of $y$ conditioned on $x$.

Historically, Laplace originally considered the maximum observed error as a measure of the correctness of a model. He soon moved to considering MAD instead. Due to his inability to exact solving both situations, he soon considered the differential MSE. Himself and Gauss (seemingly concurrently) derived the normal equations, a closed-form solution for this problem. Nowadays, solving the MAD is relatively easy by means of linear programming. As it is well known, however, linear programming does not have a closed-form solution.

From an optimization perspective, both correspond to convex functions. However, MSE is differentiable, thus, allowing for gradient-based methods, much efficient than their non-differentiable counterpart. MAD is not differentiable at $x=0$.

A further theoretical reason is that, in a bayesian setting, when assuming uniform priors of the model parameters, MSE yields normal distributed errors, which has been taken as a proof of correctness of the method. Theorists like the normal distribution because they believed it is an empirical fact, while experimentals like it because they believe it a theoretical result.

A final reason of why MSE may have had the wide acceptance it has is that it is based on the euclidean distance (in fact it is a solution of the projection problem on an euclidean banach space) which is extremely intuitive given our geometrical reality.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ (+1) for the reference to Laplace! $\endgroup$
    – Xi'an
    Commented Apr 18, 2015 at 8:42
  • 5
    $\begingroup$ "Theorists like the normal distribution because they believed it is an empirical fact, while experimentals like it because they believe it a theoretical result." -- I love it. But aren't there also direct physics applications for the Gaussian distribution? And there's also the stuff about maximum entropy distributions $\endgroup$
    – shadowtalker
    Commented Apr 18, 2015 at 13:46
  • 12
    $\begingroup$ @ssdecontrol I think the epigram is due to Henri Poincaré a little over a hundred years ago. Tout le monde y croit cependant, me disait un jour M. Lippmann, car les expérimentateurs s'imaginent que c'est un théorème de mathématiques, et les mathématiciens que c'est un fait expérimental. "Everyone is sure of this [that errors are normally distributed], Mr. Lippman told me one day, since the experimentalists believe that it is a mathematical theorem, and the mathematicians that it is an experimentally determined fact." from Calcul des probabilités (2nd ed., 1912), p. 171 $\endgroup$
    – Dilip Sarwate
    Commented Apr 18, 2015 at 21:30
  • 1
    $\begingroup$ Here is a mathematical answer. If we have a data matrix of independent variables X and a column matrix Y, then if there is a matrix b with the property Xb = Y, we have an soln. Usually we can't and we want the b that is 'closest' to an exact solution. As mathematics this is 'easy' to solve. It's the projection of Y onto the column space of X. The notions of projection and perpendicular etc, depends on the metric. The usual Euclidean L2 metric is what we are used to and it gives the least squares. The minimizing property of mse is a restatement of the fact that we have the projection. $\endgroup$
    – meh
    Commented Apr 24, 2015 at 14:03
  • 1
    $\begingroup$ I thought the priority disagreement was between Gauss and Legendre, with Legendre preceding Gauss in publishing, but Gauss preceding Legendre in informal correspondence. I'm also (vaguely) aware that Laplace's proof is considered to be superior. Any reference on these? $\endgroup$
    – PatrickT
    Commented Nov 22, 2017 at 19:38
36
$\begingroup$

As an alternative explanation, consider the following intuition:

When minimizing an error, we must decide how to penalize these errors. Indeed, the most straightforward approach to penalizing errors would be to use a linearly proportional penalty function. With such a function, each deviation from the mean is given a proportional corresponding error. Twice as far from the mean would therefore result in twice the penalty.

The more common approach is to consider a squared proportional relationship between deviations from the mean and the corresponding penalty. This will make sure that the further you are away from the mean, the proportionally more you will be penalized. Using this penalty function, outliers (far away from the mean) are deemed proportionally more informative than observations near the mean.

To give a visualisation of this, you can simply plot the penalty functions:

Comparison of MAD and MSE penalty functions

Now especially when considering the estimation of regressions (e.g. OLS), different penalty functions will yield different results. Using the linearly proportional penalty function, the regression will assign less weight to outliers than when using the squared proportional penalty function. The Median Absolute Deviation (MAD) is therefore known to be a more robust estimator. In general, it is therefore the case that a robust estimator fits most of the data points well but 'ignores' outliers. A least squares fit, in comparison, is pulled more towards the outliers. Here is a visualisation for comparison:

Comparison of OLS vs a robust estimator

Now even though OLS is pretty much the standard, different penalty functions are most certainly in use as well. As an example, you can take a look at Matlab's robustfit function which allows you to choose a different penalty (also called 'weight') function for your regression. The penalty functions include andrews, bisquare, cauchy, fair, huber, logistic, ols, talwar and welsch. Their corresponding expressions can be found on the website as well.

I hope that helps you in getting a bit more intuition for penalty functions :)

Update

If you have Matlab, I can recommend playing with Matlab's robustdemo, which was built specifically for the comparison of ordinary least squares to robust regression:

robustdemo

The demo allows you to drag individual points and immediately see the impact on both ordinary least squares and robust regression (which is perfect for teaching purposes!).

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Hmmm, so sum of squares beats sum of absolute errors. But why stop exactly at the exponent of 2? What about an exponent of 1.43423423, or 2.53459345 or e? $\endgroup$
    – Colm Bhandal
    Commented May 17, 2020 at 18:52
  • $\begingroup$ @ColmBhandal Indeed, question the conventional wisdom! Power of 2 (least squares) is mostly popular because it works out nicely mathematically, thereby making the estimation of the regression parameters possible through a closed-form simple linear algebra expression ($\widehat{\beta}=(X’X)^{-1}X’y$) instead of needing to use more difficult maximum likelihood estimation algorithms. See “Estimation Methods” on this Wikipedia for more information. $\endgroup$
    – Jean-Paul
    Commented May 17, 2020 at 19:47
  • 1
    $\begingroup$ @ColmBhandal Besides, least squares corresponds to estimating what you could call the “average” effect of the regressors (this is very clear to see when regressing a dependent variable on just an intercept: $y = \alpha + \epsilon$), whereas other loss functions correspond to other metrics. For example, absolute error corresponds more to estimating the “median” effect of the regressors. These relations help with interpreting the results, therefore making least squares so popular. $\endgroup$
    – Jean-Paul
    Commented May 17, 2020 at 19:59
  • $\begingroup$ Thanks. Glad you accept the challenge :) I guess another way of saying what I was saying would be to analyse this sentence "Using the linearly proportional penalty function, the regression will assign less weight to outliers than when using the squared proportional penalty function. The Median Absolute Deviation (MAD) is therefore known to be a more robust estimator" - this make it seem like more weight to outliers implies more robust, ad infinitum. But clearly too much weight is also an issue, right? Like calculating fourth powers would make it too sensitive? $\endgroup$
    – Colm Bhandal
    Commented May 18, 2020 at 10:30
  • 1
    $\begingroup$ @ColmBhandal You got it the wrong way around. Less weight to outliers implies a more robust estimator. That is indeed the definition of “robustness” for estimators. If you look at my original answer you’ll see that I also provide the link to a page from Matlab where different “robust” penalty functions are shown - so indeed different penalty functions are also being used to give less weight to outliers. $\endgroup$
    – Jean-Paul
    Commented May 18, 2020 at 10:47
4
$\begingroup$

In theory you could use any kind of loss function. The absolute and the squared loss functions just happen to be the most popular and the most intuitive loss functions. According to this wikipedia entry,

A common example involves estimating "location." Under typical statistical assumptions, the mean or average is the statistic for estimating location that minimizes the expected loss experienced under the squared-error loss function, while the median is the estimator that minimizes expected loss experienced under the absolute-difference loss function. Still different estimators would be optimal under other, less common circumstances.

As also explained in the wikipedia entry, the choice of the loss functions depends on how do you value deviations from your targeted object. If all deviations are equally bad for you no matter their sign, then you could use the absolute loss function. If deviations become worse for you the farther away you are from the optimum and you don't care about whether the deviation is positive or negative, then the squared loss function is your easiest choice. But if none of the above definitions of loss fit your problem at hand, because e.g. small deviations are worse for you than big deviations, then you can choose a different loss function and try to solve the minimizing problem. However the statistical properties of your solution might be hard to assess.

Improve this answer
$\endgroup$
2
  • $\begingroup$ A little detail: "If all deviations are equally bad for you no matter their sign ..": The MAD function penalizes errors linear-proportionally. Therefore errors are not 'equally bad' but 'proportionally bad' as twice the error gets twice the penalty. $\endgroup$
    – Jean-Paul
    Commented Apr 19, 2015 at 7:05
  • $\begingroup$ @Jean-Paul: You are right. I meant it that way. What I wanted to say with "equally bad" was that the gradient of the MAD is constant while the gradient for the MSE grows linearly with the error. Hence if the difference between two errors is constant no matter how far away from the optimum you are, while the same is not true for the MSE. I hope, that makes it a bit more understandable what I want to say. $\endgroup$
    – kristjan
    Commented Apr 21, 2015 at 9:33
3
$\begingroup$

As another answer has explained, minimizing squared error is not the same as minimizing absolute error.

The reason minimizing squared error is preferred is because it prevents large errors better.

Say your empolyer's payroll department accidentally pays each of a total of ten employees \$50 less than required. That's an absolute error of \$500. It's also an absolute error of \$500 if the department pays just one employee \$500 less. But it terms of squared error, it's 25000 versus 250000.

It's not always better to use squared error. If you have a data set with an extreme outlier due to a data acquisition error, minimizing squared error will pull the fit towards the extreme outlier much more than minimizing absolute error. That being said, it's -usually- better to use squared error.

Improve this answer
$\endgroup$
5
  • 4
    $\begingroup$ The reason minimizing squared error is preferred is because it prevents large errors better. - then why not cubed? $\endgroup$
    – Daniel Earwicker
    Commented Apr 18, 2015 at 10:20
  • $\begingroup$ @DanielEarwicker Cubed makes errors in the wrong direction subtractive. So it would have to be absolute cubed error, or stick to even powers. There is no really "good" reason that squared is used instead of higher powers (or, indeed, non-polynomial penalty functions). It's just easy to calculate, easy to minimize, and does the job. $\endgroup$
    – Atsby
    Commented Apr 18, 2015 at 19:28
  • 1
    $\begingroup$ Of course I should have said any higher even power! :) $\endgroup$
    – Daniel Earwicker
    Commented Apr 18, 2015 at 20:53
  • $\begingroup$ This has no upvotes (at the moment) but isn't this saying the same as the answer that (currently) has 15 votes (i.e. outliers have more effect)? Is this not getting votes because it is wrong, or because it misses some key info? Or because it does not have pretty graphs? ;-) $\endgroup$
    – Darren Cook
    Commented Apr 24, 2015 at 7:13
  • $\begingroup$ @DarrenCook I suspect the "modern" approach to stats prefers MAD over OLS, and suggesting that squared error is "usually" better earned me some downvotes. $\endgroup$
    – Atsby
    Commented Apr 24, 2015 at 21:27
-1
$\begingroup$

Short answers

  1. nope
  2. the mean has more interesting statistical properties than the median
Improve this answer
$\endgroup$
1
  • 12
    $\begingroup$ It would be great if you could qualify "more interesting statistical properties". $\endgroup$
    – Momo
    Commented Apr 18, 2015 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.