24
$\begingroup$

I have just been doing some reading on Gibbs sampling and Metropolis Hastings algorithm and have a couple of questions.

As I understand it, in the case of Gibbs sampling, if we have a large multivariate problem, we sample from the conditional distribution i.e. sample one variable while keeping all others fixed whereas in MH, we sample from the full joint distribution.

One thing the document said was that the proposed sample is always accepted in Gibbs Sampling i.e. the proposal acceptance rate is always 1. To me this seems like a big advantage as for large multivariate problems it seems that the rejection rate for MH algorithm becomes quite large. If that is indeed the case, what is the reason behind not using Gibbs Sampler all the time for generating the posterior distribution?

Improve this question
$\endgroup$
3
  • 13
    $\begingroup$ A well constructed multivariate MH proposal may greatly outperform Gibbs sampling, even when sampling from the conditionals is possible (e.g. high dimensional multivariate normal, HMC beats Gibbs by a wide margin when variables are highly correlated). This is because Gibbs sampling doesn't allow the variables to evolve jointly. It's sort of analogous to optimizing a function by iteratively optimizing the individual arguments - you may do better if you optimize wrt all of the arguments jointly rather than each one in succession, even though it is easier to do the latter. $\endgroup$
    – guy
    Commented Jun 26, 2014 at 7:10
  • $\begingroup$ Metropolis-Hastings can sample using proposals for a conditional. Are you referring to a particular kind of MH? $\endgroup$
    – Glen_b
    Commented Jun 26, 2014 at 13:17
  • 1
    $\begingroup$ Thanks for the comment. No, I was just thinking in general why Gibbs Sampler is not used more frequently. had missed the fact that the conditional distribution form has to be known a-priori for Gibbs sampling. For my current needs, it seems that a combination works best. So, use a MH step for a subset of the parameters while keeping others constant and then use Gibbs for the other subset (where the conditionals are easy to evaluate analytically). I am just starting on this, so not yet aware of various types of MH. Any advice on that is appreciated :-) $\endgroup$
    – Luca
    Commented Jun 26, 2014 at 14:05

2 Answers 2

Reset to default
15
$\begingroup$

the main rationale behind using the Metropolis-algorithm lies in the fact that you can use it even when the resulting posterior is unknown. For Gibbs-sampling you have to know the posterior-distributions which you draw variates from.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for the reply! So, with GS, the idea is that the conditionals are simpler distributions which can be sampled from quite easily while the joint distribution, while known, might be a complicated distribution which is difficult to sample from? $\endgroup$
    – Luca
    Commented Jun 26, 2014 at 6:41
  • 2
    $\begingroup$ Yes, this is true. Often times however, Gibbs-sampling and the Metropolis are being used in conjunction. So conditioning on some variables might give you a closed-form posterior, while for others this is not possible and you have to use a "Metropolis-step". In this case, you have to decide for which type of Metropolis-sampler (independence, random-walk) you go for and what kind proposal densities you use. But I guess this goes a bit too far and you should rather read into this stuff for yourself first. $\endgroup$
    – user3777456
    Commented Jun 26, 2014 at 6:52
3
$\begingroup$

Gibbs sampling breaks the curse of dimensionalality in sampling since you've broken down the (possibly high dimensional) parameter space into several low dimensional steps. Metropolis-Hastings alleviates some of the dimensionaltiy problems of generate rejection sampling techinques, but you are still sampling from a full multi-variate distribution (and deciding to accept/reject the sample) which causes the algorithm to suffer from the curse of dimensionality.

Think of it in this simplified way: it is much easier to propose an update for one variable at a time (Gibbs) than all variables simultaneously (Metropolis Hastings).

With that being said, the dimensionality of the parameters space will still affect convergence in both Gibbs and Metropolis Hastings since there are more parameters that could potentially not converge.

Gibbs is also nice because each step of the Gibbs loop may be in closed form. This is often the case in hierarchical models where each parameter is conditioned on only a few others. It is often pretty simple to contstruct your model so that each Gibbs step is in closed form (when each step is conjugate it's sometimes called "semi-conjugate"). This is nice because you're sampling from known distributions which can often be very fast.

Improve this answer
$\endgroup$
1
  • $\begingroup$ "Gibbs sampling breaks the curse of dimensionality in sampling": actually, Gibbs sampling tends to do much worse than something like Metropolis Hastings with an adaptive proposal covariance matrix. $\endgroup$
    – Cliff AB
    Commented Feb 25, 2018 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.