Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

How would one go about testing an implementation of a Bayes Factor calculation? The analogue in Frequentist hypothesis testing is fairly straightforward: generate data according to the null hypothesis, use the code to generate a p-value, repeat thousands of times with different random seeds, and look for uniformity of the computed p-values. To test an implementation of some Bayes Factor code, however, I am not sure how to proceed. Do I choose from models $M_1$ and $M_2$ with equal probability, generate the data, and test whether the $K$ values are reasonably near 1? Also is there an analogue of Frequentist power testing for Bayes Factors along the same lines (choose from the models with a biased coin flip)?

share|improve this question
I'd probably pick some $p(M_1)$ away from 0.5, say 0.1, just to make sure I hadn't done something that made my numerator and denominator always come out close to each other. Other than that, that seems like a reasonable approach to me. – jbowman Feb 24 '12 at 17:17
1  
I would not think there is a compelling reason for the Bayes factor to be close to one on average when generating from both models. It all depends on the prior distributions for both models. – Xi'an Feb 24 '12 at 21:18

1 Answer

up vote 2 down vote accepted

Here is a weak verification: if you write the Bayes factor as $$ B_{12}(x) = m_1(x)/m_2(x)\,, $$ you can simulate samples from either $m_1$ or $m_2$ (by simulating from the joint distribution under either model). For each of those samples, you can compute the average log-Bayes factor, which should be positive in the first case and negative in the second case (because it is a Kullback-Leibler divergence). Establishing those signs is not a proof everything's fine with your implementation, but at least it should hold!

share|improve this answer
thanks, this is the kind of thing I was looking for. I am a bit surprised there is no standard testing methodology, however. – shabbychef Feb 24 '12 at 21:46
BTW, should the log Bayes Factor be 'symmetric' about zero? In the sense that $\mathrm{E}\left[\log B_{12} | M1\right] = - \mathrm{E}\left[\log B_{12} | M2\right]$. – shabbychef Feb 24 '12 at 21:57
1  
1. There is no reason to develop "standard testing" for the Bayes factor. It is a quantity that reflects the ratio of evidences, not a p-value. 2. No symmetry: $\log B_{12}=-\log B_{21}$ but one side is integrated in $m_1$ and the other one in $m_2$... – Xi'an Feb 25 '12 at 6:43
Sorry, I meant "standard" in the sense of "commonly practiced", not as in "under the null". – shabbychef Feb 26 '12 at 5:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.