MCMC algorithms like Metropolis-Hastings and Gibbs sampling are ways of sampling from the joint posterior distributions.

I think I understand and can implement metropolis-hasting pretty easily--you simply choose starting points somehow, and 'walk the parameter space' randomly, guided by the posterior density and proposal density. Gibbs sampling seems very similar but more efficient since it updates only one parameter at a time, while holding the others constant, effectively walking the space in an orthogonal fashion.

In order to do this, you need the full conditional of each parameter in analytical from*. But where do these full conditionals come from? $$ P(x_1 | x_2,\ \ldots,\ x_n) = \frac{P(x_1,\ \ldots,\ x_n)}{P(x_2,\ \ldots,\ x_n)} $$ To get the denominator you need to marginalize the joint over $x_1$. That seems like a whole lot of work to do analytically if there are many parameters, and might not be tractable if the joint distribution isn't very 'nice'. I realize that if you use conjugacy throughout the model, the full conditionals may be easy, but there's got to be a better way for more general situations.

All the examples of Gibbs sampling I've seen online use toy examples (like sampling from a multivariate normal, where the conditionals are just normals themselves), and seem to dodge this issue.

* Or do you need the full conditionals in analytical form at all? How do programs like winBUGS do it?

  • 1
    Gibbs sampling is usually less efficient than Metropolis-Hastings because it goes one dimension at a time... – Xi'an Jan 12 '12 at 20:24
  • Gibbs sampling is more efficient at each individual step, but may need an awful lot more steps to converge -- and end up less efficient for a good overall result. – Lutz Prechelt Mar 19 at 13:02
up vote 7 down vote accepted

Yes, you are right, the conditional distribution needs to be found analytically, but I think there are lots of examples where the full conditional distribution is easy to find, and has a far simpler form than the joint distribution.

The intuition for this is as follows, in most "realistic" joint distributions $P(X_1,\dots,X_n)$, most of the $X_i$'s are generally conditionally independent of most of the other random variables. That is to say, some of the variables have local interactions, say $X_i$ depends on $X_{i-1}$ and $X_{i+1}$, but doesn't interact with everything, hence the conditional distributions should simplify significantly as $Pr(X_i|X_1, \dots, X_i) = Pr(X_i|X_{i-1}, X_{i+1})$

  • To add to this answer, you need not marginalize the other variables as originally stated in the question. All you need to do is to 're-organize' $Pr(X_i|X_{i-1},X_{i+1})$ such that you recognize the result as a known pdf and you are done. As long as you are able to re-organize the above everything else (i.e., all other constants, the integral in the denominator etc) will equal the appropriate constant for the pdf to integrate to 1. – user28 Nov 4 '10 at 9:15
  • 3
    They don't need to be found analytically. All full conditionals are proportional to the joint distribution, for instance. And that's all that's needed for Metropolis-Hastings. – Tristan Nov 5 '10 at 2:05
  • 1
    @Tristan of course. I am, however, talking about gibbs sampling. – gabgoh Nov 5 '10 at 6:22
  • 1
    They don't need to be found analytically for Gibbs Sampling. You just need to be able to sample, somehow, from the conditional; whether you can write down how to do this in a pretty analytic statement is not relevant. – guest Mar 2 '12 at 17:47
  • 1
    Actually, there is no need for an analytic full conditional: all that is required for Gibbs sampling to be implemented is the ability to simulate from the full conditionals. – Xi'an Nov 11 '15 at 10:28

I think you've missed the main advantage of algorithms like of Metropolis-Hastings. For Gibbs sampling, you will need to sample from the full conditionals. You are right, that is rarely easy to do. The main advantage of Metropolis-Hastings algorithms is that you can still sample one parameter at a time, but you only need to know the full conditionals up to proportionality. This is because the denominators cancel in the acceptance criteria function

The unnormalized full conditionals are often available. For instance, in your example $P(x_1 | x_2,...,x_n) \propto P(x_1,...,x_n)$, which you have. You don't need to do any integrals analytically. In most applications, a lot more will likely cancel too.

Programs like WinBugs/Jags typically take Metropolis-Hastings or slice sampling steps that only require the conditionals up to proportionality. These are easily available from the DAG. Given conjugacy, they also sometimes take straight Gibbs steps or fancy block stops.

  • 1
    Hey, thanks! I think that point about not needing the norming constant for metropolis-hastings is exactly the information I needed to make sense of all of this. I think, because the GS in WinBUGS stands for gibbs sampling, I was under the impression that gibbs superseded M-H and that the software was using gibbs exclusively. – cespinoza Nov 7 '10 at 1:54
  • 3
    The term Gibbs sampling is often used to imply that you sample one parameter at a time, even if you don't use the original idea of sampling directly from the full conditionals. All the software samples individual parameters or blocks of parameters in sequence, but the actual step type varies a lot depending on what works best. – Tristan Nov 9 '10 at 4:49
  • 2
    Almost whenever you can implement Gibbs, you can also implement Metropolis-Hastings alternatives. Higher efficiency comes from mixing both approaches. – Xi'an Jan 12 '12 at 20:25
  • This should imho be the accepted answer. – NoBackingDown Jun 11 '17 at 17:32

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.