Why is Lasso penalty equivalent to the double exponential (Laplace) prior?

Question

I have read in a number of references that the Lasso estimate for the regression parameter vector $B$ is equivalent to the posterior mode of $B$ in which the prior distribution for each $B_i$ is a double exponential distribution (also known as Laplace distribution).

I have been trying to prove this, can someone flesh out the details?

Answer 1

For simplicity let's just consider a single observation of a variable $Y$ such that $$Y|\mu, \sigma^2 \sim N(\mu, \sigma^2),$$

$\mu \sim \mbox{Laplace}(\lambda)$ and the improper prior $f(\sigma) \propto \mathbb{1}_{\sigma>0}$.

Then the joint density of $Y, \mu, \sigma^2$ is proportional to $$ f(Y, \mu, \sigma^2 | \lambda) \propto \frac{1}{\sigma}\exp \left(-\frac{(y-\mu)^2}{\sigma^2} \right) \times 2\lambda e^{-\lambda \vert \mu \vert}. $$

Taking a log and discarding terms that do not involve $\mu$, $$ \log f(Y, \mu, \sigma^2) = -\frac{1}{\sigma^2} \Vert y-\mu\Vert_2^2 -\lambda \vert \mu \vert. \quad (1)$$

Thus the maximum of (1) will be a MAP estimate and is indeed the Lasso problem after we reparametrize $\tilde \lambda = \lambda \sigma^2$.

The extension to regression is clear--replace $\mu$ with $X\beta$ in the Normal likelihood, and set the prior on $\beta$ to be a sequence of independent laplace$(\lambda)$ distributions.

Answer 2

This is obvious by inspection of the quantity the LASSO is optimizing.

Take the prior for $\beta_i$ to be independent Laplace with mean zero and some scale $\tau$.

So $p(\beta|\tau) \propto e^{-\frac{1}{2\tau} \sum_i|\beta_i|}$.

The model for the data is the usual regression assumption $y \stackrel{\text{iid}}{\sim}N(X\beta,\sigma^2)$.

$f(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma^{2}) \propto (\sigma^{2})^{-n/2} \exp\left(-\frac{1}{2{\sigma}^{2}}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)\right)$

Now minus twice the log of the posterior is of the form

$k(\sigma^2,\tau,n,p)+$ $\frac{1}{{\sigma}^{2}} (\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)+ \frac{1}{\tau} \sum_i|\beta_i|$

Let $\lambda=\sigma^2/\tau$ and we get $-2\log$-posterior of

$k(\sigma^2,\lambda,n,p)+$ $\frac{1}{{\sigma}^{2}}\left[ (\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)+ \lambda \sum_i|\beta_i|\right]$

The MAP estimator for $\beta$ minimizes the above, which minimizes

$S=(\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)+ \lambda \sum_i|\beta_i|$

So the MAP estimator for $\beta$ is LASSO.

(Here I treated $\sigma^2$ as effectively fixed but you can do other things with it and still get LASSO coming out.)

Edit: That's what I get for composing an answer off line; I didn't see a good answer was already posted by Andrew. Mine really doesn't do anything his doesn't do already. I'll leave mine for now because it gives a couple more details of the development in terms of $\beta$.