2
$\begingroup$

In Park & Casella's Bayesian LASSO model the LASSO is estimated through a scale mixture of normals:

lambda ~ gamma(... , ...) 
eta_i ~ exponential(lambda^2 / 2) 
beta_i ~ normal(0, sigma^2 * eta_i)

In their 2008 paper they remark that conditioning on the error variance is required for model convergence and for obtaining a unimodal posterior:

Conditioning on σ2 is important, because it guarantees a unimodal full posterior (see App. A). Without this, the posterior may not be unimodal, as shown by example in Appendix B. Lack of unimodality slows convergence of the Gibbs sampler and makes point estimates less meaningful.

The Bayesian Elastic Net of Li & Lin (2010) similarly has conditioning on the error variance as a vital part of the model specification. However the model is more complex so I only show the BLASSO's prior above.

The elastic net paper makes no reference at all to generalized linear models, and the LASSO paper only a very brief mention at the end. Searching for Bayesian LASSOs for GLMs mostly seems to turn up a lot of empirical Bayes estimators. However, I am interested in full Bayesian inference using JAGS or Stan for MCMC sampling. Most of the empirical Bayesian estimators I've found seem to require computing the MLEs first, but that's not an option if the model isn't full rank, and I have no idea how to use the pseudoinverse for GLMs like one would for minimum rank least squares.

So, if one is estimating a GLM using Park & Casella's model, would the lack of a variance term result in potential multimodal posteriors? My own experiments thus far seem to indicate that at the very least the end result is that the posterior distribution for the shrinkage parameters is entirely driven by the prior, which indicates to me something is wrong with a naive application of the BLASSO to GLMs.

For the record, I am very well aware that the Horseshoe is capable of better shrinkage, so I want to clarify that I am asking out of theoretical interest.

Reference: Park, T. , Casella, G. (2008) The Bayesian LASSO. Journal of the American Statistical Association, Vol. 103, No. 482. doi: 10.1198/016214508000000337 URL: https://people.eecs.berkeley.edu/~jordan/courses/260-spring09/other-readings/park-casella.pdf

Li, Qing; Lin, Nan. The Bayesian elastic net. Bayesian Anal. 5 (2010), no. 1, 151--170. doi:10.1214/10-BA506. https://projecteuclid.org/euclid.ba/1340369796

$\endgroup$
1
$\begingroup$

Here my thoughts. Hope you already found the solution as I am sort of at the same point.

Changing the Likelihood of y for GLM changes the full conditional of your betas and likely the variance. So I am not sure if beta must be conditioned on sigma in a GLM setting anyway, as the joint density of variance,beta|y might not be concave anymore... While unimodality might be nice property you wont be using a Gibbs Sampler, so multimodality might not be such an issue anymore. For your posterior predictive it also doesnt matter if its uni or multimodial...

There are already implementations of the (adapted) Park & Casellas LASSO in some packages e.g. brms

$\endgroup$
  • $\begingroup$ The LASSO implemented in brms, at least last time I looked at the generated source code, used the double exponential density directly, rather than a scale mixture. That tends to (in JAGS or openBUGS, anyway) lead to less effective sampling (higher autocorrelation, slower speed). Actually what I ended up doing is adopting the pseudovariance method discussed in "Sparsity information and regularization in the horseshoe and other shrinkage priors" (Piironen and Vehtari, 2017) and it seems to be working quite well. $\endgroup$ – BKV Jul 5 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.