3
$\begingroup$

I am trying to use MCMC to obtain the posterior probabilities of the free parameters of a model. I have tried first to leave two free parameters for my model and I was able to estimate the posterior probability for these free parameters using EMCEE. The reason that I am confident about the results is that I have tested them on simulated data and the peaks of posteriors were very close to true values.

Now, I have set four free parameters for this model but the posteriors that I have obtained is basically similar to my priors. I have chosen uniform priors for two new parameters that I left free. The posteriors for these two free parameters got uniform distributions with some spikes.

I have to mention that the characteristic property of the data is that it is very noisy. I am wondering whether I have messed up with the new likelihood that I set up for four parameters or whether MCMC can not constrain the free parameters given the property of the data (being noisy) or I need to use special type of resampling to obtain the proper posterior?

Update: Here is a file contain the information of all points have been sampled by parallel tempering method. I have used 20 temperatures, 500 walkers and for 200 times iterations. I flatted the whole resampled data for different temperatures.

In the following I computed the likelihood for the gridded values of two new parameters fixing two other parameters. The maximum value of likelihood is in agreement with the true value in the mock data. However when I leave all four parameters free and use MCMC to get the posteriors for them it fails.

enter image description here

I have another question: The quantity that I measure with the model is the orientation of each object with respect to the center for the two given free parameters of the model. However, each object has an intrinsic orientation which is the main source of error in this measurement. At the moment, I use the rms value of all the object's orientations as $\sigma$ in the likelihood but how could I include the effect of each individual intrinsic orientation of object in the likelihood in order to get a better un-biased estimator?

$\endgroup$
  • 1
    $\begingroup$ How many samples do you have? What do the MCMC diagnostics say regarding BGR and effective sample size? Is the standard deviation of the noise a parameter in your model? How does the posterior look like for that? Without knowing your model code and your likelihood I don't think this question is answerable. $\endgroup$ – Erik Oct 11 '15 at 9:53
  • $\begingroup$ @Erik I have used parallel tempering 20 temperatures, 500 walkers and 200 resampling which is in total 2000000 times. $\endgroup$ – Dalek Oct 11 '15 at 9:55
  • $\begingroup$ @Erik well when I set up the likelihood I have divided the rms of the noise. The likelihood is very complicated I could not write it down here. $\endgroup$ – Dalek Oct 11 '15 at 9:57
9
$\begingroup$

Assuming your likelihood and MCMC work correctly, which we can't know of course, and assuming your new variables are not ineffective:

  • If you increase the number of parameters under calibration with data fixed, an increase in marginal posterior uncertainty up to the point that the marginal posterior looks like the prior is quite common.

  • This happens if your data is not any more informative enough to constrain all parameters at once to a particular value (i.e. you have trade-offs in parameter space = equifinality)

  • A way to check this is to look at the correlation between parameters in the posterior - typically, you will see that while the marginal posteriors look flat or like the prior, the the multivariate posterior space is smaller than the multivariate prior space (i.e. you see correlations between the parameters that were not in the prior).

  • A further check would be to compare the prior predictive distribution to the posterior predictive distribution.

  • I'm sure there are better references, but an example of this phenomenon is in the appendix of 1, where we decrease the information in the data, and you see how marginal posteriors and correlations increase. For a moderate decrease of information in the data, you can still see the reduction of uncertainty in the posterior pair correlations, but if you further increase the information in your data, you get higher-order correlations and all looks random (it likely isn't though).

1 Hartig, F.; Dislich, C.; Wiegand, T. & Huth, A. (2014) Technical Note: Approximate Bayesian parameterization of a process-based tropical forest model. Biogeosciences, 11, 1261-1272. http://www.biogeosciences.net/11/1261/2014/bg-11-1261-2014.html

Addition: I thought for further reference, an illustration would be useful. What are are talking about is this type of correlation in posterior space:

library(psych)
par1= runif(1000,0,1)
par2 =par1 + rnorm(1000,sd = 0.05)
scatter.hist(par1,par2)

enter image description here

As one can see, the marginals look flat, but there is actually a large reduction in uncertainty in the two-dimensional space.

$\endgroup$
  • $\begingroup$ @FlorainHartig so the bottom line of your argument is that I might not be able to get the information for the new set of free parameters from the data that I have, right? or I am wondering there might be better approaches to get reliable posteriors from the data? $\endgroup$ – Dalek Oct 11 '15 at 11:04
  • $\begingroup$ I wouldn't say this - you are reducing your uncertainty in the multivariate space, as you may see if you compare prior predictive to posterior predictive. Just marginally, it doesn't look that way. $\endgroup$ – Florian Hartig Oct 11 '15 at 11:17
  • $\begingroup$ Maybe the comments at the end of github.com/florianhartig/LearningBayes/blob/master/… are helpful? $\endgroup$ – Florian Hartig Oct 11 '15 at 11:20
  • $\begingroup$ It is very nice example you have provided but alas the posteriors in 2D space of these two parameters with flat prior looks like an extended box without any specific concentration except toward one of edges. I am not sure it is very informative.. $\endgroup$ – Dalek Oct 11 '15 at 21:01
  • $\begingroup$ I'm rereading your question - I had somehow understood that you were loosing fit in 2 parameters that you were able to fit before, but on rereading it sounds now as if you have never fit those parameters that we speak about before. Are you sure they have any effect on the likelihood at all? They may be just ineffective. $\endgroup$ – Florian Hartig Oct 11 '15 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.