How would one go about testing an implementation of a Bayes Factor calculation? The analogue in Frequentist hypothesis testing is fairly straightforward: generate data according to the null hypothesis, use the code to generate a p-value, repeat thousands of times with different random seeds, and look for uniformity of the computed p-values. To test an implementation of some Bayes Factor code, however, I am not sure how to proceed. Do I choose from models $M_1$ and $M_2$ with equal probability, generate the data, and test whether the $K$ values are reasonably near 1? Also is there an analogue of Frequentist power testing for Bayes Factors along the same lines (choose from the models with a biased coin flip)?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ I'd probably pick some $p(M_1)$ away from 0.5, say 0.1, just to make sure I hadn't done something that made my numerator and denominator always come out close to each other. Other than that, that seems like a reasonable approach to me. $\endgroup$– jbowmanCommented Feb 24, 2012 at 17:17
-
1$\begingroup$ I would not think there is a compelling reason for the Bayes factor to be close to one on average when generating from both models. It all depends on the prior distributions for both models. $\endgroup$– Xi'anCommented Feb 24, 2012 at 21:18
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
Here is a weak verification: if you write the Bayes factor as $$ B_{12}(x) = m_1(x)/m_2(x)\,, $$ you can simulate samples from either $m_1$ or $m_2$ (by simulating from the joint distribution under either model). For each of those samples, you can compute the average log-Bayes factor, which should be positive in the first case and negative in the second case (because it is a Kullback-Leibler divergence). Establishing those signs is not a proof everything's fine with your implementation, but at least it should hold!
-
$\begingroup$ thanks, this is the kind of thing I was looking for. I am a bit surprised there is no standard testing methodology, however. $\endgroup$ Commented Feb 24, 2012 at 21:46
-
$\begingroup$ BTW, should the log Bayes Factor be 'symmetric' about zero? In the sense that $\mathrm{E}\left[\log B_{12} | M1\right] = - \mathrm{E}\left[\log B_{12} | M2\right]$. $\endgroup$ Commented Feb 24, 2012 at 21:57
-
1$\begingroup$ 1. There is no reason to develop "standard testing" for the Bayes factor. It is a quantity that reflects the ratio of evidences, not a p-value. 2. No symmetry: $\log B_{12}=-\log B_{21}$ but one side is integrated in $m_1$ and the other one in $m_2$... $\endgroup$– Xi'anCommented Feb 25, 2012 at 6:43
-
$\begingroup$ Sorry, I meant "standard" in the sense of "commonly practiced", not as in "under the null". $\endgroup$ Commented Feb 26, 2012 at 5:13