5
$\begingroup$

I've been looking around out of curiosity if there is a standard method equivalent to OLS that finds the coefficients satisfying the minimum absolute error criteria as opposed to squared error, but haven't found anything for $$ \min \sum_{i=1}^n ||F(x_i)-y_i||. $$ I'm not talking about a LASSO (which is a penalty on the $L_1$ norm of the coefficients). Obviously a simple custom convex optimization could do this, but just curious if I was overlooking something already commonly available by not using the right buzz words/lingo.

$\endgroup$
  • 5
    $\begingroup$ Quantile regression is a generalization of this. $\endgroup$ – whuber Dec 16 '16 at 20:15
  • 2
    $\begingroup$ There's plenty of software around that fits these models, if that's what you're trying to ask. $\endgroup$ – Glen_b Dec 16 '16 at 23:16
  • 1
    $\begingroup$ whuber's answer is exactly what I was looking for. Thanks! $\endgroup$ – JPJ Dec 17 '16 at 5:44
  • $\begingroup$ L1 regression can be formulated with li ear programming $\endgroup$ – kjetil b halvorsen Jul 19 '17 at 17:30
4
$\begingroup$

We know from intuition of the necessity of the Bessel correction that $$\arg\min_x \sum_{j=1}^n (x_j - x)^2 = \bar{x},$$ the sample mean. It similarly turns out that $$\arg\min_x \sum_{j=1}^n |x_j - x| = \mathrm{med}(x_1, \dots, x_j),$$ the sample median. Commonly in regression, we minimize the squared error, giving us estimates for the mean, but, if we were to instead minimize the absolute error, we'd get estimates for the median. Of course, when the the regression model is $y \sim \mathcal{N}(X \beta^*, \sigma^2I)$, the median is the mean so the differences between these two methods aren't too pronounced. Indeed, I've commonly seen quantile regression motivated as being useful in the presence of outliers.

An interesting history of the use of absolute error in regression is included here (in section 2):

Portnoy, S., & Koenker, R. (1997). The Gaussian Hare and the Laplacian Tortoise: Computability of Squared-Error versus Absolute-Error Estimators. Statistical Science, 12(4), 279–300.

It's discussed that the idea to use this loss was had long ago, but, due to computational intractability, it didn't gain widespread attention.

More generally, we could this is an example of quantile regression, which just installs weights into the absolute error that works for the median.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.