For ordinary least square linear regression, we have sum of residuals as zero, what about the sum of residuals for linear regression calculated using absolute loss?
2 Answers
Provided the model includes an intercept (which is implied), at least half the residuals must be non-negative and at least half must be non-positive.
This is easy to show. Suppose, on the contrary, that there are more positive residuals than non-positive residuals. Then, by increasing the intercept a tiny bit (say $\delta$) we would reduce all the positive residuals by $\delta$ while increasing all the negative residuals by $\delta,$ for a net decrease in the sum of the sizes of the residuals, thereby demonstrating the original fit could not have been the solution. The same argument applies to the case where a plurality of residuals are negative.
However, this balance between the counts of positive and negative residuals does not imply a balance in their sums -- quite the contrary. As an example, let's contemplate the simplest type of this model, where you seek a number $\mu$ that best approximates a collection of numerical data in a least absolute loss sense. The preceding argument shows this number $\mu$ must be a median of those data values. Consider, say, the dataset $(0, 1, 100).$ Its unique median is $1,$ giving residuals of $(-1,0,99).$ More than half are non-positive and more than half are non-negative. Yet, the sum of the residuals is $-1+0+99=98,$ far from zero.
This gives an accurate intuition for what's going on: minimizing absolute loss does not penalize a residual in proportion to its size; it only penalizes residuals according to whether they are positive or negative (with no penalty for zero residuals).
For more information and details, see my analysis at https://stats.stackexchange.com/a/114363/919.
-
$\begingroup$ In other words: The mean minimizes the least squares loss, and the mean is the point where the residuals sum to zero. The median minimizes the absolute loss, and the mean is the point where you have equal amount of positive and negative residuals. $\endgroup$ Jan 24, 2022 at 19:38
-
$\begingroup$ @Sextus (Understanding your last reference to "mean" was intended to be "median"): loosely, yes. To be exactly correct we must bear in mind (a) the median might not be unique and (b) the numbers of positive and negative residuals need not be equal when some residuals are zero. I have taken some care in my language here to be clear about that. $\endgroup$– whuber ♦Jan 24, 2022 at 19:50
-
$\begingroup$ On the one hand, we are minimizing the sum of absolute values of the residuals. (Are we not?) On the other hand, minimizing absolute loss does not penalize a residual in proportion to its size; it only penalizes residuals according to whether they are positive or negative (with no penalty for zero residuals). How do we reconcile these seemingly contradictory points? I suppose I should read your linked answer first, though. Perhaps I will have to delete this comment then. $\endgroup$ Jan 24, 2022 at 21:05
-
1$\begingroup$ @Richard You reconcile them by reading my argument at "this is easy to show." The uniting idea is that as we vary the parameters of the fit, we are varying the loss, which is tantamount to differentiating the loss function. When you differentiate a squared loss you obtain a linear relation for the optimum; when you differentiate an absolute loss function you are simply counting positive and negative losses. $\endgroup$– whuber ♦Jan 24, 2022 at 21:32
Not really an answer, but I would say "not much" - see my above comment. Also invoking the following code a few times suggests we also cannot sign the sum of the residuals.
As you correctly point out, OLS is the technique that exploits orthogonality of residuals and regressors, which, if we have a constant, yields a sum of residuals of zero. As that is not the case for other techniques, there also is (at least to me) no reason to expect any special properties here.
library(quantreg)
set.seed(2022)
n <- 50
X <- sort(runif(n))
beta1 <- 2
y <- X*beta1 + rnorm(n,sd=0.5)
olsreg <- lm(y~X) # OLS linear regression
ladreg <- rq(y~X) # quantile regression
sum(resid(olsreg)) # basically zero
sum(resid(ladreg)) # not close to zero
-
1$\begingroup$ @Dave, thanks for the edit, but I am not sure about the seed - my idea was that running the code a couple of times would show the reader that this or that sign may happen with the sum of the LAD residuals $\endgroup$ Jan 24, 2022 at 16:14
-
1$\begingroup$ ... Nevertheless, absolute regression residuals do enjoy "special properties." They have been exploited since this technique was first applied in the mid-18th century (before the invention of OLS!). $\endgroup$– whuber ♦Jan 24, 2022 at 16:54
-
$\begingroup$ @whuber Do you mean the first bold statement in your answer, or is there more to it? $\endgroup$– DaveJan 24, 2022 at 17:03
-
$\begingroup$ @Dave The "more to it" is covered (partly) in the link at the end of my answer. The part that was known in 1755 includes the analysis in my answer along with a comparable (but trickier) analysis of the slope in the univariate case. S. Stigler devotes a section to this in his book, The History of Statistics: see "Roger Boscovich and the Figure of the Earth." $\endgroup$– whuber ♦Jan 24, 2022 at 17:08