# Testing an implementation of Bayes Factor code

How would one go about testing an implementation of a Bayes Factor calculation? The analogue in Frequentist hypothesis testing is fairly straightforward: generate data according to the null hypothesis, use the code to generate a p-value, repeat thousands of times with different random seeds, and look for uniformity of the computed p-values. To test an implementation of some Bayes Factor code, however, I am not sure how to proceed. Do I choose from models $M_1$ and $M_2$ with equal probability, generate the data, and test whether the $K$ values are reasonably near 1? Also is there an analogue of Frequentist power testing for Bayes Factors along the same lines (choose from the models with a biased coin flip)?

• I'd probably pick some $p(M_1)$ away from 0.5, say 0.1, just to make sure I hadn't done something that made my numerator and denominator always come out close to each other. Other than that, that seems like a reasonable approach to me. – jbowman Feb 24 '12 at 17:17
• I would not think there is a compelling reason for the Bayes factor to be close to one on average when generating from both models. It all depends on the prior distributions for both models. – Xi'an Feb 24 '12 at 21:18

Here is a weak verification: if you write the Bayes factor as $$B_{12}(x) = m_1(x)/m_2(x)\,,$$ you can simulate samples from either $m_1$ or $m_2$ (by simulating from the joint distribution under either model). For each of those samples, you can compute the average log-Bayes factor, which should be positive in the first case and negative in the second case (because it is a Kullback-Leibler divergence). Establishing those signs is not a proof everything's fine with your implementation, but at least it should hold!
• BTW, should the log Bayes Factor be 'symmetric' about zero? In the sense that $\mathrm{E}\left[\log B_{12} | M1\right] = - \mathrm{E}\left[\log B_{12} | M2\right]$. – shabbychef Feb 24 '12 at 21:57
• 1. There is no reason to develop "standard testing" for the Bayes factor. It is a quantity that reflects the ratio of evidences, not a p-value. 2. No symmetry: $\log B_{12}=-\log B_{21}$ but one side is integrated in $m_1$ and the other one in $m_2$... – Xi'an Feb 25 '12 at 6:43