7
$\begingroup$

Robust linear regression from minimising the absolute deviationresults in a regression line of medians conditional on covariates, instead of means using the standard least squares methodology: Is minimizing squared error equivalent to minimizing absolute error? Why squared error is more popular than the latter?

It is useful and robust in the sense that it minimises the effect of outliers in the response variable on the fitted line. What is its analogue in the generalised linear model/maximum likelihood setting?

Improve this question
$\endgroup$
3
  • $\begingroup$ I read somewhere else that minimising the absolute deviation in linear regression is equivalent to assuming an error term that is Laplace distributed. This suggests that the likelihood function (for the robust Gaussian GLM) is obtained by assuming the response distribution is Laplace. $\endgroup$
    – Alex
    Apr 21, 2017 at 0:58
  • $\begingroup$ One reason squared error is more popular is that the objective function is quadratic and minimization, in the purely linear case, is very easy - and in the GLM case can be done easily via a Newton-like iterative procedure or any of many others. However, LAD regression requires setting up a linear program or something equivalent thereto and is slower. $\endgroup$
    – jbowman
    Apr 21, 2017 at 1:01
  • $\begingroup$ I understand that. However, I cannot find any references to LAD-like regression for GLMs $\endgroup$
    – Alex
    Apr 21, 2017 at 1:03

1 Answer 1

Reset to default
5
$\begingroup$

There's no GLM (no natural exponential family model) that corresponds to L1 (Least absolute value) regression.*

Note that if you're doing MLE then a density of form $\frac{c}{\phi}\exp(-g(\frac{y-\mathbf{x}\beta}{\phi}))$ with have log-likelihood $-n\log(\phi)-\sum_i g(\frac{y_i-\mathbf{x}_i\beta}{\phi})$.

Now maximizing likelihood with respect to the parameters in $\beta$ would correspond to minimizing $\sum_i g(\frac{y_i-\mathbf{x}_i\beta}{\phi})$.

So if you're trying to minimize $\frac{1}{\phi}\sum_i |y_i-\mathbf{x}_i\beta|=\sum_i |\frac{y_i-\mathbf{x}_i\beta}{\phi}| $... the form of $g$ and hence of the density of the errors that this will be ML for should be immediately obvious -- it's the Laplace.

It is useful and robust in the sense that it minimises the effect of outliers in the response variable on the fitted line

It provides no protection at all against influential observations, and so it's not at all robust to influential outliers, as illustrated here.

I also don't think it's quite correct to say it minimizes the effect; (ignoring the above point about influential observations -- e.g. if we're just looking at location rather than regression) it bounds the effect very nicely but if you learn about influence functions and M-estimators you'll see that there are estimators with influence functions that redescend (which L1 estimators don't), and so there's estimators of location where outliers have even less effect than they do on the median.

* Leaving aside the scale$^\dagger$ for simplicity, you just can't write $\sum_i |y_i-\mathbf{x}_i\beta|$ in the form $\sum_i -\eta(\beta)\cdot T(y_i) +A(\beta)-B(y_i)$ - the absolute value function doesn't break up like that.

$\dagger$ Actually if we only had the scale to estimate, that would be exponential family.

Improve this answer
$\endgroup$
13
  • $\begingroup$ Thanks. So there is no GLM equivalent at all of robust regression as there is no distribution from the exponential family that can be used as the response distribution. For the MLE part, where does the density you have written come from, is this some form of the density of a distribution from the exponential family? Then further on, when you say that the density of errors is the Laplace, does this mean that for all regression types (Poisson, logistic etc) you assume that the response is Laplace distributed? $\endgroup$
    – Alex
    Apr 21, 2017 at 5:22
  • 1
    $\begingroup$ You should not make the mistake of thinking that "robust regression" means L1 regression; remove that habit from your mind. Robust regression includes many things; L1 regression is considered by some people to be robust, but my answer makes it clear why many other people don't consider it robust (it can be arbitrarily affected by a single observation as my linked example clearly demonstrates! Its breakdown point is $1/n$ -- in regression terms it's really no more robust to sufficiently influential gross errors than OLS). Even if you do consider it robust it's just one of many possibilities. $\endgroup$
    – Glen_b
    Apr 21, 2017 at 5:50
  • $\begingroup$ As a result, it takes rather more than what I discuss above to make the claim "there is no GLM equivalent at all of robust regression" because we haven't looked at all of robust regression. $\endgroup$
    – Glen_b
    Apr 21, 2017 at 5:57
  • $\begingroup$ The density form that I wrote is simply a way of converting loss functions that you minimize (i.e. the $\sum_i g_i$) to a corresponding density that minimizing $\sum g$ would also be ML for (minimizing $\sum g$ maximizes $\sum -g$, which is maximizing $exp(\sum -g_i)=\prod_i exp(-g_i)$ which is now (up to scaling factors) in the form of a likelihood; so a density of that $exp(-g(z))$ form is one that has a MLE that corresponds to using $\sum_i g(z_i)$ as a loss function. In general such functions are not in the exponential family, they're more general. ... ctd $\endgroup$
    – Glen_b
    Apr 21, 2017 at 5:59
  • 1
    $\begingroup$ The quote is true. But as I have explained just because you're using a loss function that would correspond to ML for some distribution doesn't necessarily imply you have assumed that distribution to be the case. [If it is the case, then that's great for you, since you are also doing ML, and that usually means you have a bunch of nice properties including efficient estimators.] $\endgroup$
    – Glen_b
    Apr 26, 2017 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.