2
$\begingroup$

Say I have a set of $K$ models and I want to perform Bayesian model selection to see which one of those best describes my data. So I add a categorical variable with $K$ different values that indicates which model is currently tested, and I only estimate the parameters of the one model that is currently selected (Bayesian model selection).

However, as far as I understand this approach, doing this naively can create "funnels" in the probability distribution, because at any time only the parameters of a single model are constrained by the data, whereas the others can wander off freely. So the usual approach is to add pseudopriors, based on posteriors from previous runs of each model, which constrain the models currently not selected.

Now, what if I also have data from $N$ different groups (e.g. participants), so that I have a hierarchical structure. I have reasons to assume that different models best describe the data from different groups. How does this interplay with Bayesian model selection? There are some steps that are clear to me, and some that are not.

  1. The model now needs $N$ different categorical variables. One for each group in the grouping structure. These can be sampled from the same multinomial distribution (with hyperpriors, so we employ the grouping structure).
  2. Whenever a model is selected for one of the groups, parameters for that group are sampled from the hyperprior for that model.

But what about, when a model is currently not selected for one of the groups? I know I could just add pseudopriors for each model and group combination, but then I would have to estimate $N\times K$ posteriors first and I would have to include $N\times K$ pseudopriors in the final model selection step.

If I understood the approach with pseudopriors correctly, then I have the feeling that I don't really need pseudopriors in this case. But I cannot really justify this. Pseudopriors are meant to constrain the parameters when the model is not currently sampled. However, the hyperprior also constrain the parameters, so in the hierarchical setting they cannot wander off anyway. So wouldn't it be enough to just sample from the hyperpriors whenever a model is currently not selected?

$\endgroup$

1 Answer 1

3
$\begingroup$

From a theoretical perspective, the Bayesian comparison of $M$ models $\mathfrak M_m$ $m=1,...,M$ proceeds by the comparison of their posterior probabilities $$\pi(M_m|\mathbf x) \propto \pi(M_m) \int_{\Theta_m} f_m(\mathbf x|\theta) \pi_m(\theta_m)\,\text d\theta_m$$and therefore implies all priors over all models and over all model parameters. The selection of a model cannot operate from a single model perspective. Approaches based on pseudo-priors are only useful from a computational perspective, as in Carlin and Chib (1995), when one is exploring the posterior over the product space $\prod \Theta_m$ by Monte Carlo or Markov chain Monte Carlo methods.

$\endgroup$
3
  • $\begingroup$ Ok, thanks for the clarification. The relationship between the posterior and the psuedo-prior was indeed what caused my confusion. So pseudo-priors are really just a "trick" to ensure ergodicity of the MCMC model and have nothing to do with the actual model. I am not sure, however, what this implies for hierarchical models. In general, I believe, this indicates pseudo-priors are not needed, if I get convergence without them. So I should just test, and see if it converges and only add pseudo-priors if the MCMC does not converge? $\endgroup$
    – LiKao
    Commented Feb 11, 2020 at 14:16
  • $\begingroup$ Pseudo-priors are solely computational devices like auxiliary variables in other simulation methods. They are no part of the Bayesian framework per se. $\endgroup$
    – Xi'an
    Commented Feb 16, 2020 at 11:59
  • $\begingroup$ Ok, that fully clears it up. I am still not sure how to handle heterogeneity with bayesian model selection, but I will just create a new question that shows my problem in more detail. I guess I will first have to run some more simulations to see how to give a general example. $\endgroup$
    – LiKao
    Commented Feb 17, 2020 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.