A classical example of Bayesian model averaging (BMA) is the regression setup where the choice of different sets of covariates corresponds to different models $\mathcal{M}_k$, $k = 1, \ldots, K$, across which one may want to average using $$\text{pr}(\Delta \mid D) = \sum_{k = 1}^K \text{pr}(\Delta \mid \mathcal{M}_k, D)\,\text{pr}(\mathcal{M}_k \mid D)\,,\qquad (1)$$ for making inference about a quantity of interest $\Delta$ given data $D$.

I was wondering whether the following procedure would also be valid and can be framed as a BMA strategy:

Say I have a hierarchical model with a prior on some parameter but have no clear idea of how to choose one hyperparameter $\lambda$ for this prior. One option would be to try putting a hyperprior on it, then employing a fully Bayesian approach.

I was wondering however if another valid approach could be to perform $K$ separate inferences for a range of $K$ possible hyperparameters, $\lambda_k$, thus yielding $K$ different models, $\mathcal{M}_k$ and then use $(1)$ to obtain an aggregated estimate which would incorporate the uncertainty about the choice of $\lambda$.

Other (related?) question:

Imagine that I use a determinist algorithm for making inference, and that my posterior is multimodal so different initialisations of my parameters may reach different local modes and lead to different models (e.g., in a regression setup, different sets of covariates are selected). One solution which is sometimes used is to run the algorithm several times with different parameter initialisations, and then build some sort of average across the runs for the quantity of interest.

So, in that context, does a weighted average $(1)$ of the estimates obtained from several initialisations corresponds to a BMA procedure? If so, how can one understand the quantity $\text{pr}(\mathcal{M_k})$ given that the model (set of covariates included) is not specified a priori but rather obtained as a result of a given initialisation setting? (this latter question also applies to the above question where the model is a result of a given hyperparameter choice).

  • $\begingroup$ Any thoughts on this? thanks... $\endgroup$ – user79097 Mar 13 at 8:55
  • $\begingroup$ One more try...? would be really helpful to have your insights on this. thanks. $\endgroup$ – user79097 Mar 26 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.