When would one use Gibbs sampling instead of Metropolis-Hastings?

There are different kinds of MCMC algorithms:

• Metropolis-Hastings
• Gibbs
• Importance/rejection sampling (related).

Why would one use Gibbs sampling instead of Metropolis-Hastings? I suspect there are cases when inference is more tractable with Gibbs sampling than with Metropolis-Hastings, but I am not clear on the specifics.

• Off the top of my head (it's been a while, so I'm not posting this as an answer), Gibbs is faster when it works, whereas Metropolis-Hastings can cope with a wider variety of models, because it isn't confined to orthogonal steps in parameter space. Commented Nov 6, 2016 at 22:40
• You may or may not be aware that Gibbs can be viewed as an instance of Metropolis-Hastings, so you might want to clarify that you mean something like "Metropolis-Hastings with local transition distributions." Commented Nov 6, 2016 at 23:02

Firstly, let me note [somewhat pedantically] that

There are several different kinds of MCMC algorithms: Metropolis-Hastings, Gibbs, importance/rejection sampling (related).

importance and rejection sampling methods are not MCMC algorithms because they are not based on Markov chains. Actually, importance sampling does not produce a sample from the target distribution, $f$ say, but only importance weights $\omega$ say, to be used in Monte Carlo approximations of integrals related with $f$. Using those weights as probabilities to produce a sample does not lead to a proper sample from $f$, even though unbiased estimators of expectations under $f$ can be produced.

Secondly, the question

Why would someone go with Gibbs sampling instead of Metropolis-Hastings? I suspect there are cases when inference is more tractable with Gibbs sampling than with Metropolis-Hastings

does not have an answer in that a Metropolis-Hastings sampler can be almost anything, including a Gibbs sampler. I replied in rather detailed terms to an earlier and similar question. But let me add a few if redundant points here:

The primary reason why Gibbs sampling was introduced was to break the curse of dimensionality (which impacts both rejection and importance sampling) by producing a sequence of low dimension simulations that still converge to the right target. Even though the dimension of the target impacts the speed of convergence. Metropolis-Hastings samplers are designed to create a Markov chain (like Gibbs sampling) based on a proposal (like importance and rejection sampling) by correcting for the wrong density through an acceptance-rejection step. But an important point is that they are not opposed: namely, Gibbs sampling may require Metropolis-Hastings steps when facing complex if low-dimension conditional targets, while Metropolis-Hastings proposals may be built on approximations to (Gibbs) full conditionals. In a formal definition, Gibbs sampling is a special case of Metropolis-Hasting algorithm with a probability of acceptance of one. (By the way, I object to the use of inference in that quote, as I would reserve it for statistical purposes, while those samplers are numerical devices.)

Usually, Gibbs sampling [understood as running a sequence of low-dimensional conditional simulations] is favoured in settings where the decomposition into such conditionals is easy to implement and fast to run. In settings where such decompositions induce multimodality and hence a difficulty to move between modes (latent variable models like mixture models come to mind), using a more global proposal in a Metropolis-Hasting algorithm may produce a higher efficiency. But the drawback stands with choosing the proposal distribution in the Metropolis-Hasting algorithm.