# Bayesian lasso vs spike and slab

Question: What are the advantages/disadvantages of using one prior over the other for variable selection?

Suppose I have the likelihood: $$y\sim\mathcal{N}(Xw,\sigma^2I)$$ where I can put either one of the priors: $$w_i\sim \pi\delta_0+(1-\pi)\mathcal{N}(0,100)\\ \pi=0.9\,,$$ or: $$w_i\sim \exp(-\lambda|w_i|)\\ \lambda \sim \Gamma(1,1)\,.$$

I put $$\pi=0.9$$ to emphasize most of the weights are zero and a gamma prior on $$\lambda$$ to pick the 'regularizing' parameter.

However, my professor keeps insisting that the lasso version 'shrinks' the coefficients and is not actually doing proper variable selection, i.e. there is an over-shrinkage of even the relevant parameters.

I personally find implementing the Lasso version easier since I use variational Bayes. In fact the Sparse Bayesian Learning paper which effectively puts a prior of $$\frac{1}{|w_i|}$$ gives even sparser solutions.

## Reflection

Since I've left academia, I've had a chance to get some more practical experience in this area. While it is true that spike + slab methods do put a non-zero prior and hence a likelihood, the lasso based methods are (extremely) fast, and it is enough to look at the mean of the weight distribution. When you are dealing with potentially millions of parameters, this matters.

I've also gone on to recognise that my professor is an idiot who can't get past the 90s.

• Your professor is correct that it is shrinking relevant parameters, but so what? It only shrinks them to the extent that they are not contributing significantly to reducing the error. And why be focused on doing proper variable selection.. Shouldn't the focus be on reducing (test) error Dec 2, 2015 at 7:27
• For most problems yes I would agree. However, for some problems (eg. cancer detection with gene expression) it is super important to find which features are the contributing factors. p.s. I've since moved on from my postdoc since he is a moron. Machine learning ftw!!! Dec 20, 2016 at 23:14
• Spike and Slab happens to be the gold standard in variable selection and I also prefer to work with LASSO. @Sachin_ruk: the spike and slab prior can be implemented using Variational Bayes too... Sep 15, 2017 at 9:55
• @SandipanKarmakar could you post a link referring to spike and slab with Variational Bayes. Sep 17, 2017 at 11:18
• Your question merges modelling [which prior?] and implementation [variational Bayes] issues. They should be processed separately. Oct 25, 2017 at 5:17