Residual using absolute loss linear regression

Question

For ordinary least square linear regression, we have sum of residuals as zero, what about the sum of residuals for linear regression calculated using absolute loss?

Answer 1

Provided the model includes an intercept (which is implied), at least half the residuals must be non-negative and at least half must be non-positive.

This is easy to show. Suppose, on the contrary, that there are more positive residuals than non-positive residuals. Then, by increasing the intercept a tiny bit (say $\delta$) we would reduce all the positive residuals by $\delta$ while increasing all the negative residuals by $\delta,$ for a net decrease in the sum of the sizes of the residuals, thereby demonstrating the original fit could not have been the solution. The same argument applies to the case where a plurality of residuals are negative.

However, this balance between the counts of positive and negative residuals does not imply a balance in their sums -- quite the contrary. As an example, let's contemplate the simplest type of this model, where you seek a number $\mu$ that best approximates a collection of numerical data in a least absolute loss sense. The preceding argument shows this number $\mu$ must be a median of those data values. Consider, say, the dataset $(0, 1, 100).$ Its unique median is $1,$ giving residuals of $(-1,0,99).$ More than half are non-positive and more than half are non-negative. Yet, the sum of the residuals is $-1+0+99=98,$ far from zero.

This gives an accurate intuition for what's going on: minimizing absolute loss does not penalize a residual in proportion to its size; it only penalizes residuals according to whether they are positive or negative (with no penalty for zero residuals).

For more information and details, see my analysis at https://stats.stackexchange.com/a/114363/919.

Answer 2

Not really an answer, but I would say "not much" - see my above comment. Also invoking the following code a few times suggests we also cannot sign the sum of the residuals.

As you correctly point out, OLS is the technique that exploits orthogonality of residuals and regressors, which, if we have a constant, yields a sum of residuals of zero. As that is not the case for other techniques, there also is (at least to me) no reason to expect any special properties here.

library(quantreg)
set.seed(2022)
n <- 50
X <- sort(runif(n))
beta1 <- 2
y <- X*beta1 + rnorm(n,sd=0.5)
olsreg <- lm(y~X) # OLS linear regression
ladreg <- rq(y~X) # quantile regression
sum(resid(olsreg)) # basically zero
sum(resid(ladreg)) # not close to zero