21
$\begingroup$

It's well known that linear regression with an $l^2$ penalty is equivalent to finding the MAP estimate given a Gaussian prior on the coefficients. Similarly, using an $l^1$ penalty is equivalent to using a Laplace distribution as the prior.

It's not uncommon to use some weighted combination of $l^1$ and $l^2$ regularization. Can we say that this is equivalent to some prior distribution over the coefficients (intuitively, it seems that it must be)? Can we give this distribution a nice analytic form (maybe a mixture of Gaussian and Laplacian)? If not, why not?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ see this paper: tandfonline.com/doi/abs/10.1198/jasa.2011.tm09241 (If this isn't properly answered in a week or two, I'll post (more or less) a summary of that paper) $\endgroup$
    – user795305
    Commented Jun 2, 2017 at 17:10
  • 9
    $\begingroup$ I should add that any time frequentists have a penalty $pen$, a bayesian can interpret that as a (possibly improper) prior $e^{-pen}$ under a standard gaussian model. $\endgroup$
    – user795305
    Commented Jun 2, 2017 at 17:12
  • $\begingroup$ thanks, this paper and its citations answer my question perfectly! $\endgroup$
    – Michael Curry
    Commented Jun 2, 2017 at 17:15
  • 1
    $\begingroup$ Zou and Hastie 2005, which it looks like is the paper introducing the elastic net. They give an interpretation in terms of a prior. $\endgroup$
    – Michael Curry
    Commented Jun 2, 2017 at 17:25
  • 1
    $\begingroup$ Okay, cool! I think their bayesian interpretation ties into my second comment $\endgroup$
    – user795305
    Commented Jun 2, 2017 at 18:22

1 Answer 1

Reset to default
9
$\begingroup$

Ben's comment is likely sufficient, but I provide some more references one of which is from before the paper Ben referenced.

A Bayesian elastic net representation was proposed by Kyung et. al. in their Section 3.1. Although the prior for the regression coefficient $\beta$ was correct, the authors incorrectly wrote down the mixture representation.

A corrected Bayesian model for the elastic net was recently proposed by Roy and Chakraborty (their Equation 6). The authors also go on to present an appropriate Gibbs sampler to sample from the posterior distribution, and show that the Gibbs sampler converges to the stationary distribution at a geometric rate. For this reason, these references might turn out to be useful, in addition to the Hans paper.

Improve this answer
$\endgroup$
4
  • $\begingroup$ (+1) Great answer! $\endgroup$
    – user795305
    Commented Jun 5, 2017 at 2:33
  • 4
    $\begingroup$ for anyone in the future -- the papers are all worth looking at, but the Hans paper gives you some Gibbs samplers for various distributions as well as a hierarchical representation of the prior that can be translated easily to Stan. $\endgroup$
    – Michael Curry
    Commented Jun 28, 2017 at 15:16
  • $\begingroup$ Would you point out the mistakes in Kyung et. al. sec 3.1? $\endgroup$
    – Albert Chen
    Commented Nov 30, 2020 at 15:33
  • $\begingroup$ With respect to the elastic net, the errors in Section 3.1 are described in this paper on page 757. $\endgroup$
    – Greenparker
    Commented Dec 1, 2020 at 2:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.