Bayesian model averaging in the case of large number of predictors

In the context of a linear factor model, Bayesian Model Averaging (BMA) is used to obtain the posterior probability of all possible combinations of predictors. A final model is obtained as a weighted average of all models, where the weight of each model is its posterior probability.

In the case of a large number of factors K, the total number of possible combinations, i.e. models, is $2^K$. Therefore, it is almost impossible to average out over that many models.

Is there an approach to get around this difficulty?

Thanks,

• I've only ever heard of BMA being used in the context of evaluating completely disparate prediction models incorporating diverse factors and using several different approaches, such as partial least squares, neural nets, and nearest neighbor prediction models: none of which are nested. What are you hoping to achieve with such a BMA application? Alternately, there are many other high dimensional approaches which would be better fitted to your situation. Sep 18 '13 at 20:55
• There are applications of BMA for variable selection in the context of a linear factor model (ex: paper "Mixtures of g-priors for Bayesian Variable Selection"). In this context, BMA is used for selecting the best subset of predictors from a space of $K$ candidate factors. The outcome is a posterior probability associated with each possible combination of the $K$ factors. The purpose is then to get a final model as a weighted average of all those $K$ models. My question is: when K is large, how do you average over so many models (my case is: K = 14, so number of models = 16344) ? Sep 19 '13 at 15:11
• If you're interested in the best subset, how are you going about choosing that from BMA? The highest weighted model? Why not LASSO? Sep 19 '13 at 16:28
• Automated techniques such as LASSO and stepwise regression could be very dangerous, as they don't encompass model uncertainty. BMA does address the model uncertainty issue, and estimates a posterior prob. for each model rather than choosing one unique model. As an example of variable selection using BMA, take the simple example of 2 candidate factors. The possible combinations (models) are {$F_1$}, {$F_2$}, and {$F_1$,$F_2$}. BMA estimates the posterior prob. of each of the three models, and a final model is constructed as a weighted avg. of the 3 models, where weights are the posterior prob. Sep 19 '13 at 19:34
• In the context of model uncertainty, you always want to divide your data into test, training, and validation datasets regardless of the method used. This is described in the Tibs book which is free online. I think BMA implicitly does that by sampling from the model's posterior distribution to estimate posterior model probability. Anyway, it's a tough problem. In general, there's no way to do it by brute force. This might be an illuminating article: ncbi.nlm.nih.gov/pmc/articles/PMC2742495 Sep 19 '13 at 19:43

The problem you face is that the model space is too large to compute all models in reasonable time. This is a well-known problem in the Bayesian Model Averaging literature, but there are computational solutions for it, such as a MCMC algorithm. In R, there is for example the BMS package which implements such methods (see homepage). I'm sure that the above references also provide helpful ideas on how you could solve your problem.