Testing an implementation of Bayes Factor code

How would one go about testing an implementation of a Bayes Factor calculation? The analogue in Frequentist hypothesis testing is fairly straightforward: generate data according to the null hypothesis, use the code to generate a p-value, repeat thousands of times with different random seeds, and look for uniformity of the computed p-values. To test an implementation of some Bayes Factor code, however, I am not sure how to proceed. Do I choose from models $M_1$ and $M_2$ with equal probability, generate the data, and test whether the $K$ values are reasonably near 1? Also is there an analogue of Frequentist power testing for Bayes Factors along the same lines (choose from the models with a biased coin flip)?

• I'd probably pick some $p(M_1)$ away from 0.5, say 0.1, just to make sure I hadn't done something that made my numerator and denominator always come out close to each other. Other than that, that seems like a reasonable approach to me. Commented Feb 24, 2012 at 17:17
• I would not think there is a compelling reason for the Bayes factor to be close to one on average when generating from both models. It all depends on the prior distributions for both models. Commented Feb 24, 2012 at 21:18

Here is a weak verification: if you write the Bayes factor as $$B_{12}(x) = m_1(x)/m_2(x)\,,$$ you can simulate samples from either $m_1$ or $m_2$ (by simulating from the joint distribution under either model). For each of those samples, you can compute the average log-Bayes factor, which should be positive in the first case and negative in the second case (because it is a Kullback-Leibler divergence). Establishing those signs is not a proof everything's fine with your implementation, but at least it should hold!
• BTW, should the log Bayes Factor be 'symmetric' about zero? In the sense that $\mathrm{E}\left[\log B_{12} | M1\right] = - \mathrm{E}\left[\log B_{12} | M2\right]$. Commented Feb 24, 2012 at 21:57
• 1. There is no reason to develop "standard testing" for the Bayes factor. It is a quantity that reflects the ratio of evidences, not a p-value. 2. No symmetry: $\log B_{12}=-\log B_{21}$ but one side is integrated in $m_1$ and the other one in $m_2$... Commented Feb 25, 2012 at 6:43