1
$\begingroup$

Is it valid to compare mariginal probability distributions from separate Bayesian analyses to infer which scenario is most likely?

Specifically, in phylogenetic (evolutionary) analysis, if I construct three separate Bayesian analyses with different priors that affect how the phylogenetic tree looks, and each produces a different posterior mariginal probability distribution for the tree, is it valid to compare these tree likelihoods to infer which is the most likely tree topology?

Further, is it valid to use a statistical test to put a P-value on this? i.e. to say the difference between likelihoods is significantly different from zero.

Edit: I think that the results from the normal MCMC runs might not be good for comparisons, instead the mariginal likelihood estimate should be used, e.g. from http://www.beast2.org/path-sampling/:

"Say, the marginal likelihood estimate value for model 1 is X and the value for model 2 is Y, then the Bayes factor comparing model s1 and 2 is is X-Y. If this difference is positive, then the Bayes factor is in favour of model 1, if it is negative, it is in favour of model 2."

If I was to run this many times, then is it possible to calculate a p-value for the comparison? Or, is it better just to say, as worded above, that X or Y model is favoured (and the Bayes factor itself says how much it is favoured). Although this is valid for comparison, and I can now say one is favoured more than the other, I say to use a p-value as then I can say the difference between the mariginal likelihood estimate distributions is significant - so perhaps you can see my thinking, although I am aware this probably makes no sense, it would be great to hear your thoughts!

Improve this question
$\endgroup$
5
  • $\begingroup$ First MCMC is not an inference method but a Monte Carlo representation of the posterior. It is therefore possible to derive an approximation to the Bayes factor from an MCMC sample, if one stays away from the harmonic mean approximation. $\endgroup$
    – Xi'an
    Jan 23, 2019 at 5:15
  • $\begingroup$ Second I am rather reserved at using Bayes factors for comparing priors rather than sampling models, for the fundamental reason that there is a single realisation of the prior in each model: the sample, whatever its size, is associated with a single value of the parameter. Thus, there is no consistency associated with this comparison. $\endgroup$
    – Xi'an
    Jan 23, 2019 at 5:17
  • $\begingroup$ Is the prior the only thing that changes between the three analyses, or are you also using different models? $\endgroup$
    – Robin Ryder
    Jan 23, 2019 at 13:52
  • $\begingroup$ @Robin Ryder, the prior specifying the tree toplogy is the only thing that changes between the three seperate analyses $\endgroup$
    – sumade
    Jan 24, 2019 at 13:03
  • $\begingroup$ @Xi'an thanks - phylogenetic literature I have read seems to suggest that path sampling is better for comparison (I suppose this gives a distribution of mariginal likelihood estimates - but then wouldn't an MCMC run that combines multiple chains achieve this?). I think I understand the second point, but I am not sure how else to compare? I only stipulate the single parameter for tree topology - if I compare the mariginal likelihood estimates for the analysis as a whole, then I guess everything else apart from that parameter can change so there is still room to explore parameter space? $\endgroup$
    – sumade
    Jan 24, 2019 at 13:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.