Shrinkage priors

Question

I am building a Bayesian model where I to put shrinkage priors such as spike and slab, horseshoe prior, etc on some parameters for feature selection, but I am not able to decide which one is the best. What are the pros and cons of different priors so that I can choose one?

Answer 1

I don't quite know your specific use case though I can provide you with a bunch of resources so that you can make your own decision:

Michael Betancourt's case study on sparse regressions: https://betanalpha.github.io/assets/case_studies/bayes_sparse_regression.html

That is a particularly great resource because he shows the effect of different priors on the same problem. Also he provides the code for the analysis so implementing it yourself is a lot easier

And Piironen and Vehtari (2017) has a great discussion about the various choices: https://arxiv.org/abs/1707.01694

The basic points are this:

The spike and slab prior can work quite well in practice but can be sensitive to specific choices made for the prior (e.g., slab width). Additionally, the spike and slab prior can be quite computationally intensive if you have a lot of variables.

Continuous shrinkage priors (e.g., horseshoe) are computationally easier (e.g., you can implement them in Stan because they are continuous) but in the classic horseshoe there are issues such as the hyperparameter choice is arbitrary and large parameter values are not regularized. This is where the work of Piironen and Vehtari comes in.

Answer 2

You can check the following paper by Sarah Van Erp et al (2019), who discuss different shrinkage priors (below you can see table from their paper).

Those priors vary greatly in their shapes, and so amount of shrinkage they provide.

Besides discussing pros and cons of those priors, the authors describe a simulation study, where they compare performance of different priors. The results are mixed, for example, depend on if classical regularization, full Bayesian, or empirical Bayesian approach was used. What is also worth mentioning, is that in many cases the differences between different priors were rather small, with some differences when there was more features then samples.

In practice this should not makes that big difference, especially when the sample size is large. Choice would be probably rather subjective, based on how much shrinkage would you expect to be needed. You could also make some simulation study, where you would sample data from the priors^†, and check performance of the model on this data, by looking on the difference with ground truth values. In such study you would probably need to compare different amount of "false" features to be zeroed-out by shrinkage.

^{† - i.e. simulate data that is similar to what would you expect to see, assuming that you have reasonable, informative priors to achieve this.}

Van Erp, S., Oberski, D. L., & Mulder, J. (2019). Shrinkage Priors for Bayesian Penalized Regression. Journal of Mathematical Psychology, 89, 31-50. doi:10.1016/j.jmp.2018.12.004 (preprint and supplement: https://osf.io/bf5up/)