I want to compare "real" Bayesian Model Averaging (BMA) performed with the EM algorithm and information-criterion based BMA.

Which one, BIC or AIC, is a "closer" approximation to the "real" BMA?

BIC as the name is indicating? I would like to compare apples with apples.

My models are not nested with 1 parameter at minimum and 3 parameter at maximum.

  • $\begingroup$ It depends on the actual Bayesian model-for-models you want to apply model averaging to (i.e. what you mean by "real BMA" -- it depens on what your model-framework is). BIC is an asymptotic approximation to minus twice the log-posterior for what might be fairly typical formulation of how a model-situation operates but AIC can also be set up to correspond to a different form of Bayesian model choice. $\endgroup$ – Glen_b Dec 6 '16 at 5:17

I don't understand the first sentence very well, but if your asking whether you should use BIC or AIC to approximate BMA than the answer is fairly straightforward.

You would use BIC rather than AIC to approximate posterior model probabilities which can then be used as weights in BMA.

Given a set of models $M_1,M_2,…,M_n$ and respective BIC scores $B_1,B_2,…,B_n$, the posterior probability for model $j \in \{1,2,...,n\}$ can be approximated as; $$ Pr(M_j|D) \approx \frac{\exp\bigg(\frac{\mathrm{B_j}}{-2}\bigg)}{\sum_{i=1}^n \exp\bigg(\frac{\mathrm{B_i}}{-2}\bigg)} $$ Where $D$ is "the data". I briefly touch on why this is the case here. You can also see a good reference here, or take a look at the original paper where BIC is derived (Schwarz, 1978) which inevitably leads to this result.

The posterior probabilities $Pr(M_j|D)$ are used as weights in BMA, and it should be obvious that $\sum_{j=1}^n Pr(M_j|D) = 1$.

If you plan to do this in real life, I would advise using Bayes factors to avoid computational difficulties (overflow/underflow). In this case you would choose one BIC value, say the lowest one, which we will denote as $B_{*}$ then letting $\Delta_i=B_i-B_{*}$ calculate;

$$ Pr(M_j|D) \approx \frac{\exp\bigg(\frac{\Delta_j}{-2}\bigg)}{\sum_{i=1}^n \exp\bigg(\frac{\Delta_i}{-2}\bigg)} $$

This helps avoid issues of getting extremely large/small positive numbers when exponentiating the BIC values.

  • Schwarz, G. (1978). Estimating the dimension of a model. The annals of statistics, 6(2), 461-464.
| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.