Consider a simple Bayesian hierarchical model:

$y | \theta \sim P(y | \theta)$

$\theta | \phi \sim P(\theta | \phi)$

$\phi \sim P(\phi)$

I'm interested in drawing from the posterior distribution of $\phi$ using Metropolis-Hastings. Given a candidate value of $\phi$, is it acceptable to draw $\theta$ from its specified distribution and then to use this value of $\theta$ to compute $P(y | \theta)$? I understand that this is an approximation to marginalizing $\theta$, but I wanted to know if this approach is commonly used in practice (or if it has a name)?


1 Answer 1


I'm not sure exactly what you have in mind. Here are some observations.

The joint distribution for $(y,\theta,\phi)$ can always be expressed as \begin{equation} p(y,\theta,\phi) = p(y|\theta,\phi)\,p(\theta|\phi)\,p(\phi) . \end{equation} The model become hierarchical if the distribution for $y$ conditional on $\theta$ is independent of $\phi$: \begin{equation} p(y|\theta,\phi) = p(y|\theta) . \end{equation}

Consider the posterior distribution for $(\theta,\phi)$ given the hierarchical structure: \begin{equation} p(\theta,\phi|y) \propto p(y,\theta,\phi) = p(y|\theta)\,p(\theta|\phi)\,p(\phi). \end{equation} This joint posterior distribution can be characterized in terms of the full conditional distributions: \begin{align} p(\theta|y,\phi) &\propto p(y|\theta)\,p(\theta|\phi) \\ p(\phi|y,\theta) &\propto p(\theta|\phi)\,p(\phi) . \end{align} Owing to the hierarchical structure, $y$ does not appear in the right-hand side of the second expression.

A Markov chain can be constructed using the full conditional distributions as follows. Given the current state $(\theta^{(r)}, \phi^{(r)})$, the following state can be obtained via \begin{align} \phi^{(r+1)} &\sim p(\phi|\theta^{(r)}) \\ \theta^{(r+1)} &\sim p(\theta|y,\phi^{(r+1)}) . \end{align} It may not be possible to draw directly from one or both of these distributions, in which case one may use the Metropolis-Hastings algorithm.

Instead of sampling $\theta$ and $\phi$ separately, one may sample them via a single Metropolis-Hastings step. Let $(\theta',\phi')$ denote the proposal, where $(\theta',\phi') \sim q(\theta,\phi|\theta^{(r)},\phi^{(r)})$. In this case, \begin{equation} (\theta^{(r+1)},\phi^{(r+1)}) = \begin{cases} (\theta',\phi') & R \ge u \\ (\theta^{(r)},\phi^{(r)}) & \text{otherwise} \end{cases} , \end{equation} where $u \sim \textsf{Uniform}(0,1)$ and \begin{equation} R = \underbrace{\frac{p(y|\theta')\,p(\theta'|\phi')\,p(\phi')}{p(y|\theta^{(r)})\,p(\theta^{(r)}|\phi^{(r)})\,p(\phi^{(r)})}}_{\text{Metropolis}} \times \underbrace{\frac{q(\theta^{(r)},\phi^{(r)}|\theta',\phi')}{q(\theta',\phi'|\theta^{(r)},\phi^{(r)})}}_{\text{Hastings}} . \end{equation} There are two cases of particular interest. First, if the proposal distribution is symmetric, then the Hastings term equals one and disappears. Second (and perhaps this is what the OP had in mind), if $q(\theta',\phi'|\theta^{(r)},\phi^{(r)}) = p(\theta|\phi)\,p(\phi)$, then \begin{align} \phi' &\sim p(\phi) \\ \theta' &\sim p(\theta|\phi') . \end{align} and \begin{equation} R = \frac{p(y|\theta')}{p(y|\theta^{(r)})} . \end{equation}

Note that this approach to sampling doesn't really have anything to do with the hierarchical structure of the model; rather, it simply relies on using the prior for the proposal. The efficiency of this sampler depends strongly on the extent to which the prior density and the likelihood overlap.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.