I have around 20 independent (or not strongly correlated) predictor variables. And there are about 20 observations for each variable and the outcome.

I want to build a Bayesian linear regression model using these data. How to effectively select the predictors, since 20 predictors are too many?

Maybe can use information criteria (AIC, WAIC)? Do I have to try each possible combination of the model predictors?

  • $\begingroup$ I don't have too many observations, that's why I want to cut some predictor variables. $\endgroup$ – uared1776 Sep 5 '16 at 14:50
  • 1
    $\begingroup$ If you want to go full Bayesian, try spike and slab prior(en.m.wikipedia.org/wiki/Spike-and-slab_variable_selection). Or there are many papers that suggest variable selection via Zellner's g-prior. $\endgroup$ – Daeyoung Lim Sep 5 '16 at 15:31

Several types of priors for conducting variable selection have been developed in the context of linear regression models. One the most recent proposals are non-local priors, which are implemented in the R package mombf:


In your case, $2^{20}$ models are not that many (1048576), and it may be feasible to explore all of them. If you want to conduct a faster, efficient selection, you may want to have a look at either of these two papers (which are also part of the mombf package):

Variable Selection Via Gibbs Sampling. Edward I. George; Robert E. McCulloch. Journal of the American Statistical Association


J.G. Scott and J.O Berger. [Bayes and empirical Bayes multiplicity adjustment in the variable selection problem](https://projecteuclid.org/euclid.aos/1278861454). The Annals of Statistics.

  • $\begingroup$ Could you give links for the papers? $\endgroup$ – mdewey Sep 8 '16 at 16:09
  • $\begingroup$ @mdewey Done, but apparently I cannot post more than two links as a new user. Could you please give me hand with that? $\endgroup$ – Diary Sep 8 '16 at 16:18
  • $\begingroup$ I tried editing it but that does not fix it. At least the information is now there even if not as pretty as it could be. $\endgroup$ – mdewey Sep 8 '16 at 16:32

See Juho Piironen and Aki Vehtari (2017). Comparison of Bayesian predictive methods for model selection. Statistics and Computing, 27(3):711-735. http://link.springer.com/article/10.1007/s11222-016-9649-y which shows that most approaches overfit during the selection, and that projection predictive approach performs the best.

The projection predictive model selection has been implemented in projpred R package https://github.com/stan-dev/projpred which supports rstanarm models. For a quick intro see https://htmlpreview.github.io/?https://github.com/stan-dev/projpred/blob/master/vignettes/quickstart.html We have tested projection predictive model selection with many small n large n datasets up to 20k predictors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.