I am running an MCMC sampler with a model that uses Cash's C statistic for the likelihood (along with gaussian priors), which is supposed to resemble a chi square distribution in the limit of large values in observed histograms. As such, I would expect the distribution of my posterior probabilities to approximately look like a chi square:
However, the actual distribution for my model, when run with a particular dataset, seems to have one chi square distribution at a certain range of posterior probabilites, and then another, smaller chi square distribution in another range:
This is a plot of the negative of the log posterior, so the larger distribution on the left has the more probable values for the parameters. If I sample from this posterior and make mock observations with my parameter draws, I can see two distinct trends:
The downward trending model actually appears by eye to fit the data somewhat better, even though it is coming from the set of posteriors that have lower probability (i.e. from the smaller "chi square" distribution on the right in the above plot). By the way, this is a kind of projection of the data, while the model is being fit in a higher dimensional space.
I have observed similar things with several different datasets for which I am using this model. Sometimes there are even more disjoint sets of seemingly chi square distributed samples. Usually samples from the most probable portion of the posterior lead to the apparent best fits to the data but not always as appears above.
My model is far from guaranteed to account for every possible process that produced the data and so I expect there to be limits to the posterior predictions. My primary question is whether this is indicative of a serious problem with the MCMC samples and how to diagnose it. Is this something that one should not see in the distribution of posterior probabilities? Or should I not trust these results? My samples seem to be multimodal in parameter space but generally have support between modes and I think I have sampled the space quite fully. Could I be mistaken in believing this? Is that what these separated distributions in probability space are telling me?