Ergodicity of MCMC in a hierarchical model

Question

Many of the Bayesian hierarchical models that I am studying use a Markov chain as the model. These hierarchical models use different MCMC techniques to sample low-level and high-level parameters. My question is: does using different MCMC techniques for different parameters affect the ergodicity/convergence of the Markov chain in any way?

For example, in Rasmussen 2000's infinite Gaussian mixture model:

1. The variance of the Gaussian components are given an inverse Gamma prior, $p(\sigma^{-2}) \sim \mathcal{G}(\beta, w^-1)$. Within this, the $\beta$ hyper itself is given an inverse Gamma prior $\mathcal{G}(1,1)$, and the conditional posterior of $\beta$ is sampled using ARS.
   • In his subsequent paper on infinite mixtures of Gaussian Process experts, he also employs a similar hierarchical model, where he learns the kernel parameters using Hamiltonian Monte Carlo.
2. The indicator/latent variables are sampled using Gibbs sampling.
3. The Dirichlet process concentration parameter $\alpha$ is sampled using ARS.

Is there some sort of ergodicity guarantee for such models based on, say, the reversibility of the individual techniques used in each step in the chain, or is the proof a bit more subtle?

Answer 1

Supose the posterior distribution is denoted by $\pi$ defined on a subset of $\mathbb{R}^d$. Then the Markov chain that samples from this distribution is a general state space Markov chain. Here are the conditions needed for a Markov chain to be ergodic (the definitions are simplified).

• $\pi$ is the stationary distribution of the Markov chain
• the Markov chain is aperiodic, i.e., it does not get stuck in a deterministic cycle of sets in $\mathbb{R}^d$.
• the Markov chain is irreducible, i.e., in some number of steps it can potentially go from any part of the space to any other part of the space.
• the Markov chain is Harris recurrent, i.e., it does the above infinitely often

So any Markov chain constructed for general state spaces must satisfy the above properties to ensure Monte Carlo averages converge.

• If all full conditionals of distribution are well behaved (non-negative pdfs etc), then the Gibbs sampler will be ergodic, even though they may not be reversible. The right stationary distribution comes from the fact that each full conditional yields a Markov chain with stationary distribution $\pi$. Thus, a convolution (although non reversible), has stationary distribution $\pi$.
• All Metropolis-Hastings samplers are ergodic as long as the proposal distributions are well-defined. Hamiltonian Monte Carlo is a M-H sampler with an unusual proposal distribution which turns out to be well-behaved. So it is ergodic. The right stationary distribution comes from the fact that M-H are reversible with respect to $\pi$.

Ergodicity guarantees convergence of the Markov chain. However, the tricky part is trying to choose a Markov chain that converges at a fast rate. Rates of convergence of Markov chain is much more difficult to establish.

Main reference for all the above: Roberts and Rosenthal (2004)