I have 5 different posterior distributions (mcmc samples) which all estimate the same parameter beta. The 5 models are all obtained from 5 independent standardized datasets but estimate the same parameter. What I want to obtain is a single posterior distribution, combining information of my five original distribution. By instinct, following the rules of probability, I would say I can obtain a joint posterior distribution simply as the product of my 5 different posterior distributions. Is this correct, and if yes, how to do this in practice? Is there any BUGS/JAGS or R code anybody is willing to share?

  • 1
    $\begingroup$ The product of the posterior distributions corresponding to each sample is not the posterior of the pooled sample given that the prior appears 5 times (see also). Are you using the same probabilistic structure for each of the 5 samples? If you want some code, you will have to provide more information about your model and observations. $\endgroup$ – user10525 Dec 18 '12 at 17:27
  • 1
    $\begingroup$ @Procrastinator is right, you should not multiply the posteriors. Instead, put all 5 datasets and a single prior into one analysis (straightforward with BUGS/JAGS etc) and obtain a single posterior from there. $\endgroup$ – guest Dec 18 '12 at 22:45
  • $\begingroup$ I would say that if the 5 mcmc sample sets are of the same length, you can just concatenate them and this will be the combined posterior distribution. Is that correct, @Procrastinator? $\endgroup$ – Curious Dec 22 '12 at 11:53
  • $\begingroup$ @Tomas That sounds correct to me. Assuming that the underlying models are the same and the MCMC algorithms used to obtain each sample are comparable. $\endgroup$ – user10525 Dec 22 '12 at 12:03
  • 1
    $\begingroup$ @Procrastinator, thanks for explanation. So the second assumption is rather obvious for any bayesian analysis, we can skip this.. And regarding the first assumption, in that case you would be perhaps doomed anyway, even in the combined analysis proposed by guest... $\endgroup$ – Curious Dec 22 '12 at 13:01

Unfortunately, you cannot combine posterior chains in that way. From your description, what you have are independent draws from MCMC chains for the following posterior distributions:

$$p(\beta|\boldsymbol{x}_i, \pi_i),$$

where $\boldsymbol{x}_i$ is the data-set and $\pi_i$ is a prior in model $i = 1, ..., n$. You want a single posterior:

$$p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_n, \pi),$$

taken using the data from all $n$ models, where $\pi$ is the chosen prior for your analysis. Assuming that all models used the same prior that you want to use (i.e., assuming that $\pi = \pi_1 = ... = \pi_n$) gives some simplification to this problem, but it is still not enough. Unfortunately, even with this simplifying assumption, there is no function that transforms the original posteriors to the posterior you want, and similarly, no way to map the set of original MCMC chains to an appropriate MCMC chain for the posterior distribution you want.

Instead, you will need to go back to first-principles with the data sets from those models. You can either take each of the $n$ data-sets jointly, and build a single model to give an MCMC chain for the posterior from all this data, or you could do this iteratively, by updating one data-set at a time. When updating evidence in Bayesian statistics, the following recursive relationship holds:

$$p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_n, \pi) \propto p(\boldsymbol{x}_n|\beta) \cdot p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_{n-1}, \pi).$$

What this says is that the posterior given $\boldsymbol{x}_1, ..., \boldsymbol{x}_{n}$ can be obtained by using the posterior given $\boldsymbol{x}_1, ..., \boldsymbol{x}_{n-1}$ as your prior, and then updating this with the newly observed data $\boldsymbol{x}_{n}$. When using MCMC chains, this means that you would use the following looping procedure.

For $i = 1, ..., n-1$ do the following:

  1. Take the posterior MCMC chain from the distribution $p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_i, \pi)$.

  2. Use this as a chain of prior values for $\beta$. Input this into your MCMC calculations updating for the new data-set $\boldsymbol{x}_{i+1}$.

  3. This gives you a posterior MCMC chain from $p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_{i+1}, \pi)$. Generate the new posterior MCMC chain.

Running through this iterative process gives you a posterior MCMC chain for the distribution $p(\beta|\boldsymbol{x}_1, ..., \boldsymbol{x}_n, \pi)$, which is of interest.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.