# Test for comparing log likelihoods that include error terms

Suppose I have two models with likelihoods calculated through thermodynamic integration. thermodynamic_integration_log_evidence in PTSampler returns both an estimate of the integral and an error term for it, which the docs say arises from sampling at a finite number of temperatures.

If I was confident in the likelihoods $L_1, L_2$ for my models $M_1,M_2$ then I could just compute the Bayes factor $L_1/L_2$ (or $exp(LL_1-LL_2)$ for log likelihoods) to see how many times more likely is $M_1$ than $M_2$.

But as $L_1, L_2$ come with corresponding errors $\sigma_{L1}, \sigma_{L2}$ how can I take these into account in the comparison?

EDIT

I have realised that, if nothing else, I can use monte carlo to draw lots of samples of $(L_1,L_2)$, then for each sample compute the probability $P(M_1 is\ the\ best|M_1\ or\ M_2\ is\ the\ best)$ - given equal priors of $1/2$ for $M_1$ and $M_2$ I can just normalize the likelihoods so they sum to 1 then take the mean normalized $L_1/2$ as my final probability that $M_1$ is the better model... I think (please correct if wrong)!

All this however depends on knowing the distribution of likelihoods given by thermodynamic integration - I'm assuming normal but might also be wrong about that?

• I can't decide whether it is an interesting question, or just one that I can't understand. Maybe you can clarify a bit, or help me understand the question. To my knowledge the Likelihood of any given combination of model and data is just a number. In fact, on itself it is a meaningless number. Depending on what you are doing, you can go to great lengths and make a probabilistic interpretation possible. To my understanding, the errors you obtain from the method are methodical errors -- and not errors to the model. But I'm not familiar with the particular method you have been using. May 24, 2018 at 16:14
• As I understand it (could be wrong) likelihood alone is not quite meaningless - if L=1 then the model fits the data perfectly for all parameter values allowed by the priors. In practice (as in this question) you can interpret two likelihoods using the Bayes factor to give a relative likelihood en.wikipedia.org/wiki/Bayes_factor May 24, 2018 at 21:39
• You are correct, this is a methodical error - but that still affects reliability of model comparison May 24, 2018 at 22:03
• See my edit above May 24, 2018 at 22:03
• If you do Bayesian model comparison, then afaik there is no way to take any kind of uncertainty into account. The important passage in the mentioned wikipedia article is regardless of whether these models are correct. If you want/need something like uncertainty accounted for, then you'll have to move to the realm of hypothesis testing. And of course you are right, that you can interpret Likelihood values under certain circumstances. But getting a prior right can be close to impossible; which is what I meant by the modifying statement. May 25, 2018 at 12:05

• Ok, so how would you state the actual ratio of the probabilities, given uncertainty in the method? is my monte carlo idea correct? I accept that there are issues with getting priors correct, but I find it unlikely that there is no way to include any uncertainty at all. For example say you had $LL_1 = -2000 \pm 1000, LL_2 = -2003 \pm 1000$. Comparing liklihoods without error would give Bayes factor $K=e^3=20$ suggesting "strong" evidence in favour of M1, but intuitively that seems implausible with so much methodological error. May 29, 2018 at 11:31