I am trying to formally compare the distribution of the likelihood values generated using two different models with marginal posterior values of the parameters obtained using MCMC in order to assess whether one model is good enough approximation to the other one. Both models have the same number of parameters, the only difference between these models is that one model has a restriction on one of the proposed values of parameters that it must be positive.

I should also add that the mean of the posterior median of the likelihood values from various data sets each run using both set of model assumptions is close under both models.

I have plotted the distribution of the log-likelihood values against each other and their values appear close to each other for the entire distribution . However, I am not sure how to formally compare them. I don't think that AIC, BIC are appropriate since I am not comparing models with different number of parameters to test for the best one and I am not sure about DIC.

I was thinking of using KL-divergence. Following the discussion in another question, I could convert the log- likelihood values to a probability but I am not sure that this would get me what I want since the Kl-divergence value is not a hypothesis test as such. Perhaps, my question is ill-posed and /or what I have done graphically to compare the distributions is sufficient but I wanted other peoples' opinions on this.

Oh and while using a posterior predictive distribution may be a way around this. I have several simulated data sets all of the same form so it would not be reasonable to do this several times.

  • $\begingroup$ If we know that the two distributions are different, a test will mostly show that we have not sampled enough samples. I would just use the graphics. $\endgroup$ – svendvn Apr 12 '17 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.