I am learning regession, and I dont understand why do we need to square in residual sum of squares. whats wrong with just use residual sum to represent as the error value? What is the benefit of squaring the residual?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ See also stats.stackexchange.com/questions/118/… $\endgroup$– TimCommented Sep 27, 2016 at 14:30
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
5
Squaring the residuals changes the shape of the regularization function. In particular, large errors are penalized more with the square of the error. Imagine two cases, one where you have one point with an error of 0 and another with an error of 10, versus another case where you have two points with an error of 5. The linear error function will treat both of these as having equal sum of residuals, while the squared error will penalize the case with the large error more.
With a squared residual, your solution will prefer more small errors to having any large errors. The linear residual is indifferent, not caring whether the total error is all coming from one sample or spread out as a sum of many tiny errors.
You could also raise the error to a higher power to penalize large errors even more. Summing the tenth power of the residuals, for example, would likely result in a solution that has small errors for most points, but no large errors for any one point.
answered Sep 27, 2016 at 14:32
-
$\begingroup$ it is always more clear with a lovely example. thank you so much. $\endgroup$ Commented Sep 27, 2016 at 14:34
-
$\begingroup$ In even simpler terms, does this allow a linear regression to not be overly influenced by an outlier? $\endgroup$ Commented Feb 14, 2022 at 5:46
-
1$\begingroup$ @PrithviBoinpally Higher powers of the error term (square, cube, etc.) make the fit more sensitive to outliers. Rather than fitting the main cloud of points and missing the outlier by a wide margin, higher power error terms will result in a fit that misses all the points by a smaller amount. Without the square, eleven errors of 1 unit is worse than one error of 10 units. With the square, one error of 10 units is worse than ninety-nine errors of 1 unit. $\endgroup$ Commented Feb 14, 2022 at 14:26
-
$\begingroup$ Hmm, I'm still a bit confused, would that mean that as the error term increases (^2->^3 etc.) the fit would get closer and closer to hitting outliers? Meaning, it views the big errors caused by outliers as more severe so it tries to avoid allowing such an error? $\endgroup$ Commented Mar 9, 2022 at 13:56
-
1$\begingroup$ @PrithviBoinpally Yes, with higher powers on the error, big errors get penalized more, so the solution tries to avoid them. Having a big error on certain samples counts for more the higher the power is - with higher powers, the solution will prefer to miss many points by a small amount, rather than missing a few points by a large amount (the outliers). $\endgroup$ Commented Mar 9, 2022 at 15:00
$\begingroup$
$\endgroup$
4
@ocram's answer is good, but one point I'd add is the connection between least squares and maximum likelihood estimation. If we have a regression model of the form $y_i = \beta_0 + \sum_{j=1}^{p} \beta_j x_{ij} + \epsilon_i$ where the $\epsilon_i$ are independent normal$(0, \sigma^2)$ random variables then the likelihood function becomes
$$ \mathcal{L}(\beta) = \frac{1}{\sigma^n \sqrt{2 \pi}^n} \exp \left ( - \frac{\sum_{i=1}^{n} (y_i - \beta_0 - \sum_{j=1}^{p} \beta_j x_{ij})^2}{2 \sigma^2} \right ) . $$
If we want to maximize this as a function of $\beta$ that's equivalent to minimizing $\sum_{i=1}^{n} (y_i - \beta_0 - \sum_{j=1}^{p} \beta_j x_{ij})^2$, and this is nothing but the least squares criterion.
It's also interesting to note that means themselves are least squares estimates in the univariate case, so if we agree that means are good things to look at then least squares makes sense.
-
$\begingroup$ Doesn't the pdf of normal distribution follow from the fact that we use squared residuals? We construct the likelihood based on the pdf of the normal distribution. If the answer to my question is yes (which I think), of course maximizing the likelihood is the same as minimizing SSR, since it follows by construction (right)? If the answer is no, could you please explain why (or should I ask a seperate question)? $\endgroup$– Marcel10Commented Sep 27, 2016 at 14:48
-
$\begingroup$ You could of course derive it that way but you don't have to. No where in the central limit theorem for instance do we insist that the square function needs to appear in the density of the limit, so it appears even when we don't require it. $\endgroup$– dsaxtonCommented Sep 27, 2016 at 14:55
-
-
$\begingroup$ It seems that the square function arises out of the Pythagorean theorem, which might give us more comfort that it's a natural thing to look at. $\endgroup$– dsaxtonCommented Sep 27, 2016 at 15:12
$\begingroup$
$\endgroup$
5
If you do not square, a negative residual (below the line) can offset the impact of a positive residual (above the line). Squaring is a remedy. Taking the absolute values of the residuals provides an alternative. But squaring is much easier to handle from a mathematical point of view (cf. derivatives).
-
$\begingroup$ so true. square will not give negative value. for do calculation in computer, i believe taking the absolute value is also the same as doing square! $\endgroup$ Commented Sep 27, 2016 at 14:25
-
4$\begingroup$ @BryanFok it is not the same. Square provides a larger penalty for large residuals than taking the absolute value. $\endgroup$ Commented Sep 27, 2016 at 14:28
-
$\begingroup$ @MatthewGunn That is very good point too. as we want to give penalty to prediction which is far off from our prediction. In this case, can we use power of 3 instead of square? :) $\endgroup$ Commented Sep 27, 2016 at 14:31
-
1$\begingroup$ @BryanFok No, a negative value to the power of 3 is also a negative value... So see the answer of ocram $\endgroup$– Marcel10Commented Sep 27, 2016 at 14:39
-
1$\begingroup$ @BryanFok To the power of 'an even number larger than 2' can be used, but is (as far as I know) very unconventional $\endgroup$– Marcel10Commented Sep 27, 2016 at 14:40