0
$\begingroup$

I am running an MCMC sampler with a model that uses Cash's C statistic for the likelihood (along with gaussian priors), which is supposed to resemble a chi square distribution in the limit of large values in observed histograms. As such, I would expect the distribution of my posterior probabilities to approximately look like a chi square:

distribution of chi square

However, the actual distribution for my model, when run with a particular dataset, seems to have one chi square distribution at a certain range of posterior probabilites, and then another, smaller chi square distribution in another range:

double chi square?

This is a plot of the negative of the log posterior, so the larger distribution on the left has the more probable values for the parameters. If I sample from this posterior and make mock observations with my parameter draws, I can see two distinct trends:

posterior draw observations

The downward trending model actually appears by eye to fit the data somewhat better, even though it is coming from the set of posteriors that have lower probability (i.e. from the smaller "chi square" distribution on the right in the above plot). By the way, this is a kind of projection of the data, while the model is being fit in a higher dimensional space.

I have observed similar things with several different datasets for which I am using this model. Sometimes there are even more disjoint sets of seemingly chi square distributed samples. Usually samples from the most probable portion of the posterior lead to the apparent best fits to the data but not always as appears above.

My model is far from guaranteed to account for every possible process that produced the data and so I expect there to be limits to the posterior predictions. My primary question is whether this is indicative of a serious problem with the MCMC samples and how to diagnose it. Is this something that one should not see in the distribution of posterior probabilities? Or should I not trust these results? My samples seem to be multimodal in parameter space but generally have support between modes and I think I have sampled the space quite fully. Could I be mistaken in believing this? Is that what these separated distributions in probability space are telling me?

$\endgroup$
  • $\begingroup$ Can you explain how you come to the conclusion that the posterior should look like a $\chi^2$ distribution? $\endgroup$ – Greenparker Sep 5 '18 at 7:10
  • $\begingroup$ If I were evaluating a chi square statistic, where all the assumptions of that statistic were satisfied by my data and model, I'd expect to see the values distributed like a chi square. Since the statistic I use in the likelihood is supposed to approach a chi square in some limit and my priors are quite broad, I imagine that I would see something similar if my data were well behaved and the model were reasonable. Am I wrong in that? I am mostly concerned with whether this kind of multiplicity of peaks in my posterior values is indicative of some problem or if I can safely use them $\endgroup$ – kiliantics Sep 5 '18 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.