The goal of gibbs sampling is to sample the joint distribution when this latter has not an analytic expression, by deriving the conditional distribution of each variable. So it is supposed that the conditional distribution of each variable has an analytic expression. What if they don't have it? Which method could I use?


Gibbs sampling is most efficient if you can sample directly from the conditional distributions. A common example is a Normal-Gamma mixture distribution, e.g. $$ p(\mu,\lambda)=p(\mu|\lambda)p(\lambda)=Normal(\mu; \mu0, \lambda^{-1})Gamma(\lambda; a_0,b_0) $$ It's easy to sample from a Gamma or Normal distribution (which we can do directly, via their inverse CDFs), but harder to sample from the Normal-Gamma joint distribution. Gibbs sampling is useful here because direct sampling (which we can do from the conditionals but no the joint) is normally more efficient than rejection-based methods (such as Metropolis-Hastings or similar techniques).

So we don't use Gibbs sampling because the joint distribution isn't available. In fact, if all the conditionals are available, then by definition the joint must be too (because you can calculate the joint from the conditionals), so usually this is not the reason. Rather, when we use Gibbs we normally do it because it's faster or more convenient than the alternatives. For instance, in the above example, we do know the joint: it's a Normal-Gamma, and we could use any number of techniques to sample from this distribution. It's just that Gibbs is faster in this case because it can benefit from direct sampling.

On the other hand, if you cannot sample directly from any of the conditionals, and thus would have to use a rejection-based sampler (like Metropolis-Hastings) for each conditional sample, then Gibbs is usually not a good choice and will be slower than sampling from the joint.

Gibbs sampling does not necessarily require that you have an analytic expression for the conditionals, just that you can sample from them. Of course in practice, sampling from the conditionals often requires that you do know their PDFs (at least up to a constant of proportionality). But in some cases, you may be able simulate the random process that gives rise to your variables, without explicitly knowing their distributions, and then you can still do (Gibbs) sampling that way.

If you are neither able to sample via the known conditional distributions of your variables, nor through a simulations of their (conditional) generative processes, then I don't see how you could still do Gibbs sampling. If you do have access to the joint, then you could use any number of other MCMC techniques instead (with Metropolis Hastings being one of the most widely-used ones).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.