6
$\begingroup$

Robust linear regression from minimising the absolute deviationresults in a regression line of medians conditional on covariates, instead of means using the standard least squares methodology: Is minimizing squared error equivalent to minimizing absolute error? Why squared error is more popular than the latter?

It is useful and robust in the sense that it minimises the effect of outliers in the response variable on the fitted line. What is its analogue in the generalised linear model/maximum likelihood setting?

$\endgroup$
  • $\begingroup$ I read somewhere else that minimising the absolute deviation in linear regression is equivalent to assuming an error term that is Laplace distributed. This suggests that the likelihood function (for the robust Gaussian GLM) is obtained by assuming the response distribution is Laplace. $\endgroup$ – Alex Apr 21 '17 at 0:58
  • $\begingroup$ One reason squared error is more popular is that the objective function is quadratic and minimization, in the purely linear case, is very easy - and in the GLM case can be done easily via a Newton-like iterative procedure or any of many others. However, LAD regression requires setting up a linear program or something equivalent thereto and is slower. $\endgroup$ – jbowman Apr 21 '17 at 1:01
  • $\begingroup$ I understand that. However, I cannot find any references to LAD-like regression for GLMs $\endgroup$ – Alex Apr 21 '17 at 1:03
3
$\begingroup$

There's no GLM (no natural exponential family model) that corresponds to L1 (Least absolute value) regression.*

Note that if you're doing MLE then a density of form $\frac{c}{\phi}\exp(-g(\frac{y-\mathbf{x}\beta}{\phi}))$ with have log-likelihood $-n\log(\phi)-\sum_i g(\frac{y_i-\mathbf{x}_i\beta}{\phi})$.

Now maximizing likelihood with respect to the parameters in $\beta$ would correspond to minimizing $\sum_i g(\frac{y_i-\mathbf{x}_i\beta}{\phi})$.

So if you're trying to minimize $\frac{1}{\phi}\sum_i |y_i-\mathbf{x}_i\beta|=\sum_i |\frac{y_i-\mathbf{x}_i\beta}{\phi}| $... the form of $g$ and hence of the density of the errors that this will be ML for should be immediately obvious -- it's the Laplace.

It is useful and robust in the sense that it minimises the effect of outliers in the response variable on the fitted line

It provides no protection at all against influential observations, and so it's not at all robust to influential outliers, as illustrated here.


I also don't think it's quite correct to say it minimizes the effect; (ignoring the above point about influential observations -- e.g. if we're just looking at location rather than regression) it bounds the effect very nicely but if you learn about influence functions and M-estimators you'll see that there are estimators with influence functions that redescend (which L1 estimators don't), and so there's estimators of location where outliers have even less effect than they do on the median.


* Leaving aside the scale$^\dagger$ for simplicity, you just can't write $\sum_i |y_i-\mathbf{x}_i\beta|$ in the form $\sum_i -\eta(\beta)\cdot T(y_i) +A(\beta)-B(y_i)$ - the absolute value function doesn't break up like that.

$\dagger$ Actually if we only had the scale to estimate, that would be exponential family.

$\endgroup$
  • $\begingroup$ Thanks. So there is no GLM equivalent at all of robust regression as there is no distribution from the exponential family that can be used as the response distribution. For the MLE part, where does the density you have written come from, is this some form of the density of a distribution from the exponential family? Then further on, when you say that the density of errors is the Laplace, does this mean that for all regression types (Poisson, logistic etc) you assume that the response is Laplace distributed? $\endgroup$ – Alex Apr 21 '17 at 5:22
  • 1
    $\begingroup$ You should not make the mistake of thinking that "robust regression" means L1 regression; remove that habit from your mind. Robust regression includes many things; L1 regression is considered by some people to be robust, but my answer makes it clear why many other people don't consider it robust (it can be arbitrarily affected by a single observation as my linked example clearly demonstrates! Its breakdown point is $1/n$ -- in regression terms it's really no more robust to sufficiently influential gross errors than OLS). Even if you do consider it robust it's just one of many possibilities. $\endgroup$ – Glen_b Apr 21 '17 at 5:50
  • $\begingroup$ As a result, it takes rather more than what I discuss above to make the claim "there is no GLM equivalent at all of robust regression" because we haven't looked at all of robust regression. $\endgroup$ – Glen_b Apr 21 '17 at 5:57
  • $\begingroup$ The density form that I wrote is simply a way of converting loss functions that you minimize (i.e. the $\sum_i g_i$) to a corresponding density that minimizing $\sum g$ would also be ML for (minimizing $\sum g$ maximizes $\sum -g$, which is maximizing $exp(\sum -g_i)=\prod_i exp(-g_i)$ which is now (up to scaling factors) in the form of a likelihood; so a density of that $exp(-g(z))$ form is one that has a MLE that corresponds to using $\sum_i g(z_i)$ as a loss function. In general such functions are not in the exponential family, they're more general. ... ctd $\endgroup$ – Glen_b Apr 21 '17 at 5:59
  • $\begingroup$ ctd ... however, the exponential family is of the form $f_{X}(x\mid \theta )=\exp \left(\eta (\theta )\cdot T(x)-A(\theta )+B(x)\right)$, so they fall into that very, very general framework (we can see that there's a simple "loss function" for any member of the exponential family we can immediately write down, $\sum_i -\eta(\theta)\cdot T(y_i) +A(\theta)-B(y_i))$ (note that the predictors appear in $\eta$, and we can drop the term in $B$ without affecting the fit). Your last question makes no sense to me. You were asking which density would have L1 regression as ML. It's the Laplace. ...ctd $\endgroup$ – Glen_b Apr 21 '17 at 6:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.