18
$\begingroup$

It's well known that linear regression with an $l^2$ penalty is equivalent to finding the MAP estimate given a Gaussian prior on the coefficients. Similarly, using an $l^1$ penalty is equivalent to using a Laplace distribution as the prior.

It's not uncommon to use some weighted combination of $l^1$ and $l^2$ regularization. Can we say that this is equivalent to some prior distribution over the coefficients (intuitively, it seems that it must be)? Can we give this distribution a nice analytic form (maybe a mixture of Gaussian and Laplacian)? If not, why not?

$\endgroup$
  • 2
    $\begingroup$ see this paper: tandfonline.com/doi/abs/10.1198/jasa.2011.tm09241 (If this isn't properly answered in a week or two, I'll post (more or less) a summary of that paper) $\endgroup$ – user795305 Jun 2 '17 at 17:10
  • 9
    $\begingroup$ I should add that any time frequentists have a penalty $pen$, a bayesian can interpret that as a (possibly improper) prior $e^{-pen}$ under a standard gaussian model. $\endgroup$ – user795305 Jun 2 '17 at 17:12
  • $\begingroup$ thanks, this paper and its citations answer my question perfectly! $\endgroup$ – Michael Curry Jun 2 '17 at 17:15
  • 1
    $\begingroup$ Zou and Hastie 2005, which it looks like is the paper introducing the elastic net. They give an interpretation in terms of a prior. $\endgroup$ – Michael Curry Jun 2 '17 at 17:25
  • 1
    $\begingroup$ Okay, cool! I think their bayesian interpretation ties into my second comment $\endgroup$ – user795305 Jun 2 '17 at 18:22
7
$\begingroup$

Ben's comment is likely sufficient, but I provide some more references one of which is from before the paper Ben referenced.

A Bayesian elastic net representation was proposed by Kyung et. al. in their Section 3.1. Although the prior for the regression coefficient $\beta$ was correct, the authors incorrectly wrote down the mixture representation.

A corrected Bayesian model for the elastic net was recently proposed by Roy and Chakraborty (their Equation 6). The authors also go on to present an appropriate Gibbs sampler to sample from the posterior distribution, and show that the Gibbs sampler converges to the stationary distribution at a geometric rate. For this reason, these references might turn out to be useful, in addition to the Hans paper.

$\endgroup$
  • $\begingroup$ (+1) Great answer! $\endgroup$ – user795305 Jun 5 '17 at 2:33
  • 2
    $\begingroup$ for anyone in the future -- the papers are all worth looking at, but the Hans paper gives you some Gibbs samplers for various distributions as well as a hierarchical representation of the prior that can be translated easily to Stan. $\endgroup$ – Michael Curry Jun 28 '17 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.