17
$\begingroup$

MCMC algorithms like Metropolis-Hastings and Gibbs sampling are ways of sampling from the joint posterior distributions.

I think I understand and can implement metropolis-hasting pretty easily--you simply choose starting points somehow, and 'walk the parameter space' randomly, guided by the posterior density and proposal density. Gibbs sampling seems very similar but more efficient since it updates only one parameter at a time, while holding the others constant, effectively walking the space in an orthogonal fashion.

In order to do this, you need the full conditional of each parameter in analytical from*. But where do these full conditionals come from? $$ P(x_1 | x_2,\ \ldots,\ x_n) = \frac{P(x_1,\ \ldots,\ x_n)}{P(x_2,\ \ldots,\ x_n)} $$ To get the denominator you need to marginalize the joint over $x_1$. That seems like a whole lot of work to do analytically if there are many parameters, and might not be tractable if the joint distribution isn't very 'nice'. I realize that if you use conjugacy throughout the model, the full conditionals may be easy, but there's got to be a better way for more general situations.

All the examples of Gibbs sampling I've seen online use toy examples (like sampling from a multivariate normal, where the conditionals are just normals themselves), and seem to dodge this issue.

* Or do you need the full conditionals in analytical form at all? How do programs like winBUGS do it?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Gibbs sampling is usually less efficient than Metropolis-Hastings because it goes one dimension at a time... $\endgroup$
    – Xi'an
    Commented Jan 12, 2012 at 20:24
  • $\begingroup$ Gibbs sampling is more efficient at each individual step, but may need an awful lot more steps to converge -- and end up less efficient for a good overall result. $\endgroup$
    – Lutz Prechelt
    Commented Mar 19, 2018 at 13:02

2 Answers 2

Reset to default
7
$\begingroup$

Yes, you are right, the conditional distribution needs to be found analytically, but I think there are lots of examples where the full conditional distribution is easy to find, and has a far simpler form than the joint distribution.

The intuition for this is as follows, in most "realistic" joint distributions $P(X_1,\dots,X_n)$, most of the $X_i$'s are generally conditionally independent of most of the other random variables. That is to say, some of the variables have local interactions, say $X_i$ depends on $X_{i-1}$ and $X_{i+1}$, but doesn't interact with everything, hence the conditional distributions should simplify significantly as $Pr(X_i|X_1, \dots, X_i) = Pr(X_i|X_{i-1}, X_{i+1})$

Improve this answer
$\endgroup$
5
  • $\begingroup$ To add to this answer, you need not marginalize the other variables as originally stated in the question. All you need to do is to 're-organize' $Pr(X_i|X_{i-1},X_{i+1})$ such that you recognize the result as a known pdf and you are done. As long as you are able to re-organize the above everything else (i.e., all other constants, the integral in the denominator etc) will equal the appropriate constant for the pdf to integrate to 1. $\endgroup$
    – user28
    Commented Nov 4, 2010 at 9:15
  • 3
    $\begingroup$ They don't need to be found analytically. All full conditionals are proportional to the joint distribution, for instance. And that's all that's needed for Metropolis-Hastings. $\endgroup$
    – Tristan
    Commented Nov 5, 2010 at 2:05
  • 1
    $\begingroup$ @Tristan of course. I am, however, talking about gibbs sampling. $\endgroup$
    – gabgoh
    Commented Nov 5, 2010 at 6:22
  • 1
    $\begingroup$ They don't need to be found analytically for Gibbs Sampling. You just need to be able to sample, somehow, from the conditional; whether you can write down how to do this in a pretty analytic statement is not relevant. $\endgroup$
    – guest
    Commented Mar 2, 2012 at 17:47
  • 1
    $\begingroup$ Actually, there is no need for an analytic full conditional: all that is required for Gibbs sampling to be implemented is the ability to simulate from the full conditionals. $\endgroup$
    – Xi'an
    Commented Nov 11, 2015 at 10:28
11
$\begingroup$

I think you've missed the main advantage of algorithms like of Metropolis-Hastings. For Gibbs sampling, you will need to sample from the full conditionals. You are right, that is rarely easy to do. The main advantage of Metropolis-Hastings algorithms is that you can still sample one parameter at a time, but you only need to know the full conditionals up to proportionality. This is because the denominators cancel in the acceptance criteria function

The unnormalized full conditionals are often available. For instance, in your example $P(x_1 | x_2,...,x_n) \propto P(x_1,...,x_n)$, which you have. You don't need to do any integrals analytically. In most applications, a lot more will likely cancel too.

Programs like WinBugs/Jags typically take Metropolis-Hastings or slice sampling steps that only require the conditionals up to proportionality. These are easily available from the DAG. Given conjugacy, they also sometimes take straight Gibbs steps or fancy block stops.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Hey, thanks! I think that point about not needing the norming constant for metropolis-hastings is exactly the information I needed to make sense of all of this. I think, because the GS in WinBUGS stands for gibbs sampling, I was under the impression that gibbs superseded M-H and that the software was using gibbs exclusively. $\endgroup$
    – cespinoza
    Commented Nov 7, 2010 at 1:54
  • 3
    $\begingroup$ The term Gibbs sampling is often used to imply that you sample one parameter at a time, even if you don't use the original idea of sampling directly from the full conditionals. All the software samples individual parameters or blocks of parameters in sequence, but the actual step type varies a lot depending on what works best. $\endgroup$
    – Tristan
    Commented Nov 9, 2010 at 4:49
  • 2
    $\begingroup$ Almost whenever you can implement Gibbs, you can also implement Metropolis-Hastings alternatives. Higher efficiency comes from mixing both approaches. $\endgroup$
    – Xi'an
    Commented Jan 12, 2012 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.