36
$\begingroup$

I have read in a number of references that the Lasso estimate for the regression parameter vector $B$ is equivalent to the posterior mode of $B$ in which the prior distribution for each $B_i$ is a double exponential distribution (also known as Laplace distribution).

I have been trying to prove this, can someone flesh out the details?

$\endgroup$
2

2 Answers 2

41
$\begingroup$

For simplicity let's just consider a single observation of a variable $Y$ such that $$Y|\mu, \sigma^2 \sim N(\mu, \sigma^2),$$

$\mu \sim \mbox{Laplace}(\lambda)$ and the improper prior $f(\sigma) \propto \mathbb{1}_{\sigma>0}$.

Then the joint density of $Y, \mu, \sigma^2$ is proportional to $$ f(Y, \mu, \sigma^2 | \lambda) \propto \frac{1}{\sigma}\exp \left(-\frac{(y-\mu)^2}{\sigma^2} \right) \times 2\lambda e^{-\lambda \vert \mu \vert}. $$

Taking a log and discarding terms that do not involve $\mu$, $$ \log f(Y, \mu, \sigma^2) = -\frac{1}{\sigma^2} \Vert y-\mu\Vert_2^2 -\lambda \vert \mu \vert. \quad (1)$$

Thus the maximum of (1) will be a MAP estimate and is indeed the Lasso problem after we reparametrize $\tilde \lambda = \lambda \sigma^2$.

The extension to regression is clear--replace $\mu$ with $X\beta$ in the Normal likelihood, and set the prior on $\beta$ to be a sequence of independent laplace$(\lambda)$ distributions.

$\endgroup$
0
31
$\begingroup$

This is obvious by inspection of the quantity the LASSO is optimizing.

Take the prior for $\beta_i$ to be independent Laplace with mean zero and some scale $\tau$.

So $p(\beta|\tau) \propto e^{-\frac{1}{2\tau} \sum_i|\beta_i|}$.

The model for the data is the usual regression assumption $y \stackrel{\text{iid}}{\sim}N(X\beta,\sigma^2)$.

$f(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma^{2}) \propto (\sigma^{2})^{-n/2} \exp\left(-\frac{1}{2{\sigma}^{2}}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)\right)$

Now minus twice the log of the posterior is of the form

$k(\sigma^2,\tau,n,p)+$ $\frac{1}{{\sigma}^{2}} (\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)+ \frac{1}{\tau} \sum_i|\beta_i|$

Let $\lambda=\sigma^2/\tau$ and we get $-2\log$-posterior of

$k(\sigma^2,\lambda,n,p)+$ $\frac{1}{{\sigma}^{2}}\left[ (\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)+ \lambda \sum_i|\beta_i|\right]$

The MAP estimator for $\beta$ minimizes the above, which minimizes

$S=(\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)+ \lambda \sum_i|\beta_i|$

So the MAP estimator for $\beta$ is LASSO.

(Here I treated $\sigma^2$ as effectively fixed but you can do other things with it and still get LASSO coming out.)

Edit: That's what I get for composing an answer off line; I didn't see a good answer was already posted by Andrew. Mine really doesn't do anything his doesn't do already. I'll leave mine for now because it gives a couple more details of the development in terms of $\beta$.

$\endgroup$
5
  • 1
    $\begingroup$ There seems to be a difference in your answer and Andrew's. Your answer has the correct form of the regularizer: $\lambda \|\beta\|_1$, whereas Andrew has $\lambda |\mu|$, where in linear regression, we get $\mu=X\beta$. $\endgroup$
    – Alex R.
    Nov 20, 2015 at 18:47
  • 2
    $\begingroup$ @AlexR I think you're misinterpreting the μ in Andrew's answer. The μ there corresponds to a $\beta_0$ in a regression with only an intercept, not to $X\beta$ in a multiple regression; the same argument follows for the larger case (note the parallels with my answer) but it's easier to follow in the simple case. Andrew's answer is essentially right but doesn't connect all the dots to the original question, leaving a small amount for the reader to fill in. I think our answers are consistent (up to some minor differences relating to σ that can be accounted for) and that he fully deserved the tick $\endgroup$
    – Glen_b
    Nov 20, 2015 at 22:53
  • $\begingroup$ Sorry, but could you tell me what is the $k(\sigma^2, \lambda, n, p)$ here? $\endgroup$
    – heyzude
    Sep 20, 2023 at 23:37
  • $\begingroup$ Note that we didn't start with an equality, but a proportional relationship -- the normalizing constant for the posterior is omitted. It's simply a numeric constant (it's a function of those listed parameters only, not of the data, so when those values are all fixed, it's just some number). When we take logs of both sides, we move from there being a missing normalizing constant to a missing constant-shift. ... ctd $\endgroup$
    – Glen_b
    Sep 21, 2023 at 1:24
  • $\begingroup$ ctd ... We also accumulate additional scale factors in the derivation (so the scaling constant on the original scale is the original normalizing constant times some additional constant terms), and again on the log-scale it's all just a constant "shift" up or down. We don't really care about its exact value for the present purpose, so it all gets lumped into a single constant term, which I called k. At the end we might then work out what its value "should" have been (either algebraically or numerically), if that becomes necessary (but it is rarely needed). $\endgroup$
    – Glen_b
    Sep 21, 2023 at 1:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.