I have a couple of question regarding robust regression with M-estimators, such as Huber estimator or Tukey biweight estimator:

  1. Is it possible/common to combine these with regularization terms, such as those used in ridge and lasso regression?
  2. Given that M-estimators originate from maximum likelihood, do they occur in bayesian framework? (and how)

Both extensions seem obvious to me, but searching the internet for either produced surprisingly few hits, so I expect that there are some caveats.

  1. Yes, it's possible, since you can write M-estimation in terms of a loss function (the rho function), to which you can add a penalty, reducing it to another optimization problem. However some M-estimators can have multiple modes on the likelihood, which L1 or L2 regularization won't necessarily remove.

  2. While M-estimation arises from likelihood ideas, it doesn't always correspond to maximizing a likelihood. Some M-estimators do. The Huber does, for example. However, some don't - like the Tukey biweight (indeed any M-estimator whose $\psi$-function cuts off/goes to 0 at some finite value can't be MLE for any density). The psi function can redescend and you may still have it correspond to likelihood (e.g. it does with the Cauchy) but for the $\psi$-function to correspond to a density, it can't descend too quickly -- or the corresponding density would not be integrable.

    When it does correspond to a density, you can then cast the M-estimation problem into a Bayesian framework.

    It might perhaps be possible to get it in a Bayesian framework in some other circumstances but I can't think of any that don't involve doing something odd like imposing a proper prior which might perhaps combine with some M-estimators to emulate a different improper prior times a valid likelihood (not corresponding to the M-estimator you started with). I wouldn't count that sort of thing myself

| cite | improve this answer | |
  • $\begingroup$ Thank you for these explanations! $\endgroup$ – Vadim Dec 4 '19 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.