7
$\begingroup$

As I was working my way through my book about statistics, I came across the topic of linear regression. During the chapter the author begins with explaining that you want to minimize the residuals in order to make your y = a + bx as good a fit as possible: I do understand this, but halfway the chapter all of a sudden the residuals change into sum squares of residuals. Why is this done? I have been googling, but couldn't find the right answer. Who would like to help me understand why the sums of squares of the residuals is used instead of just the sums of residuals?

Kind regards, Bas

$\endgroup$
  • $\begingroup$ You need to combine the residuals in some criterion to minimise them overall. Sums of squares is an algebraically convenient way to do it, and has some arguably useful properties. $\endgroup$ – Glen_b -Reinstate Monica Jan 27 '13 at 23:48
  • 2
    $\begingroup$ stats.stackexchange.com/questions/46019/… is essentially a duplicate. $\endgroup$ – whuber Jan 30 '13 at 15:58
10
$\begingroup$

The sums of residuals will always be 0, so that won't work.

A more interesting question is why use sum of squared residuals vs. sum of absolute value of residuals. This penalizes large residuals more than small ones. I believe the reason this is done is because the math works out more easily and, back before computers, it was much easier to estimate the regression using squared residuals. Nowadays, this reason no longer applies mean absolute deviation regression is, indeed, possible. It is one form of robust regression.

$\endgroup$
  • $\begingroup$ @Peter Flom, I thought the extra penalisation of large residuals was an advantage of using the squared values? Since larger residuals are less likely to be random deviations. You seem to think it is a disadvantage? Do you know any reasons for that? $\endgroup$ – Marloes Jan 27 '13 at 16:51
  • 4
    $\begingroup$ @kwanti If you have outliers then using the squared values typically causes the fit to adjust more to the outliers than if you used the absolute value of the residuals. So minimizing the sum of the absolute residuals could be seen as more resistant to outliers. $\endgroup$ – Dason Jan 27 '13 at 17:36
  • 1
    $\begingroup$ The sum of residuals is not necessarily zero. An intercept must be present in the model for this to be true, in general. $\endgroup$ – cardinal Jan 27 '13 at 20:22
  • 1
    $\begingroup$ @kwanti It is not an advantage or a disadvantage, necessarily; it is a difference. MAD regression is more resistant to outliers. That can be good or bad. $\endgroup$ – Peter Flom - Reinstate Monica Jan 27 '13 at 22:49
  • 2
    $\begingroup$ Another difference between least squares and MAD is that least squares gives a single unique answer while there are cases using MAD that will result in infinitely many lines that result in the same minimum MAD value. $\endgroup$ – Greg Snow Jan 30 '13 at 17:28
9
$\begingroup$

Another way to motivate the squared residuals is by making the often reasonable assumption that the residuals are Gaussian distributed. In other words, we assume that $$y = ax + b + \varepsilon$$ for Gaussian noise $\varepsilon$. In this case, the log-likelihood of the parameters $a, b$ is given by $$\log p(y \mid x, a, b) = \log \mathcal{N}(y; ax + b, 1) = -\frac{1}{2} (y - [a + bx])^2 + \text{const},$$ so that maximizing the likelihood amounts to minimizing the squared residuals.

If the noise $\varepsilon$ was Laplace distributed, the absolute value of residuals would be more appropriate. But because of the central limit theorem, Gaussian noise is much more common.

$\endgroup$
  • 3
    $\begingroup$ This is correct but it's interesting that from a historical point of view it's a perfectly circular argument: the Gaussian distribution first arose because Gauss found it is the distribution for which the MLE minimizes the sum of squared residuals! $\endgroup$ – whuber Jan 30 '13 at 19:01
  • 1
    $\begingroup$ @whuber, yes, but as the Gaussian distribution has clearly its place in nature (i.e. has many other "natural properties"), you can build on it if you tell the story the other way :-) ("natural properties" -> Gaussian distr. -> squared residuals) $\endgroup$ – Curious Jan 30 '13 at 22:35
6
$\begingroup$

Good answers, but maybe I can give a more intuitive answer. Suppose you are fitting a linear model, represented here by a straight line parameterized by a slope and intercept.

Each residual is a spring between each data point and the line, and it is trying to pull the line to itself. enter image description here
A sensible thing to do is find the slope and intercept that minimizes the energy of the system. The energy in each spring (i.e. residual) is proportional to its length squared. So what the system does is minimize the sum of the squared residuals, i.e. minimize the sum of energy in the springs.

$\endgroup$
4
$\begingroup$

In addition to the points made by Peter Flom and Lucas, a reason for minimizing the sum of squared residuals is the Gauss-Markov Theorem. This says that if the assumptions of classical linear regression are met, then the ordinary least squares estimator is more efficient than any other linear unbiased estimator. 'More efficient' implies that the variances of the estimated coefficients are lower; in other words, the estimated coefficients are more precise. The theorem holds even if the residuals do not have a normal or Gaussian distribution.

However, the theorem is not relevant to the specific comparison between minimizing the sum of absolute values and minimizing the sum of squares since the former is not a linear estimator. See this table contrasting their properties, showing advantages of least squares as stability in response to small changes in data, and always having a single solution.

$\endgroup$
  • $\begingroup$ "More efficient implies that the variances are lower" - I think you are going in circles, because variance is based on the sum of squares. Had you been using some other measure based on absolute values instead, it would possibly favor the absolute values. $\endgroup$ – Curious Jan 29 '13 at 10:18
  • $\begingroup$ @Tomas Thank you, I see the point, it leads back to the question why, or should, we want estimates of coefficients to be precise as measured by minimum variance, rather than some other measure of precision. Having said that, minimum variance is one popular measure of precision, so the G-M Theorem helps to explain why OLS regression is widely used. $\endgroup$ – Adam Bailey Jan 29 '13 at 10:55
  • $\begingroup$ There is a lot of good material on these issues in stats.stackexchange.com/questions/46019/… and stats.stackexchange.com/questions/118/…. $\endgroup$ – Adam Bailey Jan 29 '13 at 12:54
  • $\begingroup$ Gaus-Markov implies that no other method has a smaller variance. If you want to minimize the variance, use least squares. I don't see where it is "going in circles" as much as a thing that makes sense. To complete the answer to the question asked, one would say "We use the squares, instead of the absolutes, because we want to minimize the variance. The GM Theorem shows us that using the squares (doing OLS) is indeed the method which minimizes the variance". It is a perfectly good explanation for using the squares (edit: given all assumptions etc.) $\endgroup$ – IMA Jan 29 '13 at 13:00
  • $\begingroup$ These comments appear in places to use "variance" in two senses: the variance of the residuals and the variance of the estimates. Among linear estimators (not "all" estimators, pace IMA), least squares minimizes the estimation variance. It is a theorem that the estimation variance is "based on the sum of squares" of the residuals, provided the estimator is linear. @Tomas If the estimator is not linear, then the estimation variance is not proportional to the sum of squares of the residuals, so there's nothing circular about Adam's statement--and he is clear about the assumptions. $\endgroup$ – whuber Jan 30 '13 at 18:59
3
$\begingroup$

This is more a response to @PeterFlom's comment on my comment, but it is too big to fit in a comment (and does relate to the original question).

Here is some R code to show a case where there are multiple lines that all give the same minimum MAD/SAD values.

The first part of the example is clearly contrived data to demonstrate, but the end includes more of a random element to demonstrate that the general concept will still hold in some more realistic cases.

x <- rep(1:10, each=2)
y <- x/10 + 0:1
plot(x,y)

sad <- function(x,y,coef) { # mad is sad/n
    yhat <- coef[1] + coef[2]*x
    resid <- y - yhat
    sum( abs( resid ) )
}

library(quantreg)
fit0 <- rq( y~x )
abline(fit0)

fit1 <- lm( y~x, subset= c(1,20) )
fit2 <- lm( y~x, subset= c(2,19) )
fit3 <- lm( y~x, subset= c(2,20) )
fit4 <- lm( y~x, subset= c(1,19) )

fit5.coef <- c(0.5, 1/10)

abline(fit1)
abline(fit2)
abline(fit3)
abline(fit4)
abline(fit5.coef)
for (i in seq( -0.5, 0.5, by=0.1 ) ) {
    abline( fit5.coef + c(i,0) )
}

tmp1 <- seq( coef(fit1)[1], coef(fit2)[1], len=10 )
tmp2 <- seq( coef(fit1)[2], coef(fit2)[2], len=10 )

for (i in seq_along(tmp1) ) {
    abline( tmp1[i], tmp2[i] )
}

sad(x,y, coef(fit0))
sad(x,y, coef(fit1))
sad(x,y, coef(fit2))
sad(x,y, coef(fit3))
sad(x,y, coef(fit4))
sad(x,y, fit5.coef )

for (i in seq( -0.5, 0.5, by=0.1 ) ) {
    print(sad(x,y, fit5.coef + c(i,0) ))
}

for (i in seq_along(tmp1) ) {
    print(sad(x,y, c(tmp1[i], tmp2[i]) ) )
}

set.seed(1)
y2 <- y + rnorm(20,0,0.25)
plot(x,y2)
fitnew <- rq(y2~x)  # note the still non-unique warning
abline(fitnew)
abline(coef(fitnew) + c(.1,0))
abline(coef(fitnew) + c(0, 0.01) )
sad( x,y2, coef(fitnew) )
sad( x,y2, coef(fitnew)+c(.1,0))
sad( x,y2, coef(fitnew)+c(0,0.01))
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.