22
$\begingroup$

We are all familiar with the notion, well documented in the literature, that LASSO optimization (for sake of simplicity confine attention here to the case of linear regression) $$ {\rm loss} = \| y - X \beta \|_2^2 + \lambda \| \beta \|_1 $$ is equivalent to the linear model with Gaussian errors in which the parameters are given the Laplace prior
$$ \exp(-\lambda \| \beta \|_1 ) $$ We are also aware that the higher one sets the tuning parameter, $\lambda $, the larger the portion of parameters get set to zero. This being said, I have the following thought question:

Consider that from the Bayesian point of view we can calculate the posterior probability that, say, the non-zero parameter estimates lie in any given collection of intervals and the parameters set to zero by the LASSO are equal to zero. What has me confused is, given that the Laplace prior is continuous (in fact absolutely continuous) then how can there be any mass on any set that is a product of intervals and singletons at $\{0\}$?

Improve this question
$\endgroup$
7
  • 9
    $\begingroup$ What makes you think that the posterior isn't also a continuous pdf? The fact that the maximum of the posterior happens to occur at a point that happens to have lots of 0 components doesn't mean by itself that the posterior isn't a continuous pdf. $\endgroup$
    – Brian Borchers
    Dec 21, 2015 at 23:37
  • $\begingroup$ The posterior is a continuous PDF. Viewed as constrained maximum likelihood estimation, if we imagine repeated draws from the same data distribution when the true model has zeros at multiple regression coefficients and the tuning constant is large enough then the CMLE will always have the same components set to zero and the non-zero parameters will spread out into corresponding confidence intervals. From the bayesian perspective this is equivalent to having a positive probability for such sets. My question is how can this be so for a continuous distribution. $\endgroup$
    – Grant Izmirlian
    Dec 22, 2015 at 0:30
  • 2
    $\begingroup$ The CLME solution coincides with the MAP estimate. There's really nothing more to be said. $\endgroup$
    – Sycorax ♦
    Dec 22, 2015 at 2:13
  • 3
    $\begingroup$ The CMLE solution isn't a sample from the posterior. $\endgroup$
    – Brian Borchers
    Dec 22, 2015 at 6:15
  • 2
    $\begingroup$ There is no contradiction because the posterior does not put mass on sets of lower dimension. $\endgroup$
    – Xi'an
    Jan 9, 2016 at 14:26

1 Answer 1

Reset to default
7
$\begingroup$

Like all the comments above, the Bayesian interpretation of LASSO is not taking the expected value of the posterior distribution, which is what you would want to do if you were a purist. If that would be the case, then you would be right that there is very small chance that the posterior would be zero given the data.

In reality, the Bayesian interpretation of LASSO is taking the MAP (Maximum A Posteriori) estimator of the posterior. It sounds like you are familiar, but for anyone who is not, this is basically Bayesian Maximum Likelihood, where you use the value that corresponds to the maximum probability of occurrence (or the mode) as your estimator for the parameters in LASSO. Since the distribution increases exponentially until zero from the negative direction and falls off exponentially in the positive direction, unless your data strongly suggests the beta is some other significant value, the maximum value of value of your posterior is likely to be 0.

Long story short, your intuition seems to be based on the mean of the posterior, but the Bayesian interpretation of LASSO is based on taking the mode of the posterior.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.