# Test for comparing log likelihoods that include error terms

Suppose I have two models with likelihoods calculated through thermodynamic integration. thermodynamic_integration_log_evidence in PTSampler returns both an estimate of the integral and an error term for it, which the docs say arises from sampling at a finite number of temperatures.

If I was confident in the likelihoods $L_1, L_2$ for my models $M_1,M_2$ then I could just compute the Bayes factor $L_1/L_2$ (or $exp(LL_1-LL_2)$ for log likelihoods) to see how many times more likely is $M_1$ than $M_2$.

But as $L_1, L_2$ come with corresponding errors $\sigma_{L1}, \sigma_{L2}$ how can I take these into account in the comparison?

EDIT

I have realised that, if nothing else, I can use monte carlo to draw lots of samples of $(L_1,L_2)$, then for each sample compute the probability $P(M_1 is\ the\ best|M_1\ or\ M_2\ is\ the\ best)$ - given equal priors of $1/2$ for $M_1$ and $M_2$ I can just normalize the likelihoods so they sum to 1 then take the mean normalized $L_1/2$ as my final probability that $M_1$ is the better model... I think (please correct if wrong)!

All this however depends on knowing the distribution of likelihoods given by thermodynamic integration - I'm assuming normal but might also be wrong about that?

• I can't decide whether it is an interesting question, or just one that I can't understand. Maybe you can clarify a bit, or help me understand the question. To my knowledge the Likelihood of any given combination of model and data is just a number. In fact, on itself it is a meaningless number. Depending on what you are doing, you can go to great lengths and make a probabilistic interpretation possible. To my understanding, the errors you obtain from the method are methodical errors -- and not errors to the model. But I'm not familiar with the particular method you have been using. May 24, 2018 at 16:14
• As I understand it (could be wrong) likelihood alone is not quite meaningless - if L=1 then the model fits the data perfectly for all parameter values allowed by the priors. In practice (as in this question) you can interpret two likelihoods using the Bayes factor to give a relative likelihood en.wikipedia.org/wiki/Bayes_factor May 24, 2018 at 21:39
• You are correct, this is a methodical error - but that still affects reliability of model comparison May 24, 2018 at 22:03
• See my edit above May 24, 2018 at 22:03
• If you do Bayesian model comparison, then afaik there is no way to take any kind of uncertainty into account. The important passage in the mentioned wikipedia article is regardless of whether these models are correct. If you want/need something like uncertainty accounted for, then you'll have to move to the realm of hypothesis testing. And of course you are right, that you can interpret Likelihood values under certain circumstances. But getting a prior right can be close to impossible; which is what I meant by the modifying statement. May 25, 2018 at 12:05

## 1 Answer

Taking this from the comment section to the answer part:

If you are doing Bayesian model comparison, then there is no way to include any uncertainties. The comparison will tell you, which model describes the data better.

But, as mentioned in the wikipedia article (https://en.wikipedia.org/wiki/Bayes_factor), this doesn't mean that either model "fits" the data or even that either is correct.

If the uncertainty is to be taken into account, then hypothesis testing can be used to determine which fits better. But again, this will not give any clue about whether either (actually: if any!) model is correct.

That being said, you cannot determine a probability for a certain model to be correct. Nothing in the likelihood method will do this for you or enable you to do it. If you use it for comparison -- as in the question --, then you can determine which model is more probable. If (!) you get the priors right, then you can state the actual ratio of the probabilities. But nothing more.

• Ok, so how would you state the actual ratio of the probabilities, given uncertainty in the method? is my monte carlo idea correct? I accept that there are issues with getting priors correct, but I find it unlikely that there is no way to include any uncertainty at all. For example say you had $LL_1 = -2000 \pm 1000, LL_2 = -2003 \pm 1000$. Comparing liklihoods without error would give Bayes factor $K=e^3=20$ suggesting "strong" evidence in favour of M1, but intuitively that seems implausible with so much methodological error. May 29, 2018 at 11:31
• The practical solution is to minimize the numerical uncertainties -- make the error smaller. This means you'd have to run the PTSampler with much higher statistics. I can't tell conclusively if your MC approach is also possible. It is important to realize that the value and its error have entirely different meanings. I understand your uneasiness with the result of your example likelihood comparison. But you cannot treat the errors with "usual error propagation", because the likelihood value is not fluctuating statistically. Your method of calculating is inaccurate. May 29, 2018 at 11:52