2
$\begingroup$

I have around 20 independent (or not strongly correlated) predictor variables. And there are about 20 observations for each variable and the outcome.

I want to build a Bayesian linear regression model using these data. How to effectively select the predictors, since 20 predictors are too many?

Maybe can use information criteria (AIC, WAIC)? Do I have to try each possible combination of the model predictors?

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't have too many observations, that's why I want to cut some predictor variables. $\endgroup$
    – uared1776
    Commented Sep 5, 2016 at 14:50
  • 1
    $\begingroup$ If you want to go full Bayesian, try spike and slab prior(en.m.wikipedia.org/wiki/Spike-and-slab_variable_selection). Or there are many papers that suggest variable selection via Zellner's g-prior. $\endgroup$
    – Daeyoung
    Commented Sep 5, 2016 at 15:31

2 Answers 2

Reset to default
1
$\begingroup$

Several types of priors for conducting variable selection have been developed in the context of linear regression models. One the most recent proposals are non-local priors, which are implemented in the R package mombf:

https://cran.r-project.org/web/packages/mombf/index.html

In your case, $2^{20}$ models are not that many (1048576), and it may be feasible to explore all of them. If you want to conduct a faster, efficient selection, you may want to have a look at either of these two papers (which are also part of the mombf package):

Variable Selection Via Gibbs Sampling. Edward I. George; Robert E. McCulloch. Journal of the American Statistical Association

and

J.G. Scott and J.O Berger. [Bayes and empirical Bayes multiplicity adjustment in the variable selection problem](https://projecteuclid.org/euclid.aos/1278861454). The Annals of Statistics.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Could you give links for the papers? $\endgroup$
    – mdewey
    Commented Sep 8, 2016 at 16:09
  • $\begingroup$ @mdewey Done, but apparently I cannot post more than two links as a new user. Could you please give me hand with that? $\endgroup$
    – Diary
    Commented Sep 8, 2016 at 16:18
  • $\begingroup$ I tried editing it but that does not fix it. At least the information is now there even if not as pretty as it could be. $\endgroup$
    – mdewey
    Commented Sep 8, 2016 at 16:32
5
$\begingroup$

See Juho Piironen and Aki Vehtari (2017). Comparison of Bayesian predictive methods for model selection. Statistics and Computing, 27(3):711-735. http://link.springer.com/article/10.1007/s11222-016-9649-y which shows that most approaches overfit during the selection, and that projection predictive approach performs the best.

The projection predictive model selection has been implemented in projpred R package https://github.com/stan-dev/projpred which supports rstanarm models. For a quick intro see https://htmlpreview.github.io/?https://github.com/stan-dev/projpred/blob/master/vignettes/quickstart.html We have tested projection predictive model selection with many small n large n datasets up to 20k predictors.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.