Implementation of Metropolis-Hastings with conditional posterior

Question

I'm trying to understand how to estimate the parameter vector $\mathbf{\theta} = (\theta_1,\theta_2, \theta_3)$ of a model using the MH algorithm.

I am given a joint posterior density:

$p(\mathbf{\theta}, \alpha | y) \propto \alpha^n e^{\left(-\alpha f(\mathbf{\theta})\right)}$

I am also given two conditional posterior distributions:

1. $p_{1}(\alpha|\theta,y) \propto IG(n,f(\theta))$
2. $p_{2}(\theta|y) \propto f(\theta)^{-n}$

Where $f(\theta)$ is always the same function (cannot sample from it directly) which depends on $(\theta_1, \theta_2, \theta_3)$. Also, the parameter $\alpha$ is of no interest.

This is how I'm implementing the algorithm (the proposal $q$ is a multivariate Gaussian distribution):

$\theta^* \sim \mathcal{N}(\theta^{i-1}, \Sigma)$

$p = \min \left( \frac{p_2(\theta^*|y) q(\theta^{i-1}|\theta^*)}{p_2(\theta^{i-1}|y) q(\theta^*|\theta^{i-1})},1 \right) $

accept $\theta^*$ with probability $p$.

Where $q(\theta^{i-1}|\theta^*)$ is the density of the multivariate Gaussian distribution centered at $\theta^*$.

Question : am I supposed to be using the conditional posterior density $p_2$ for this step or should I use the joint posterior density? What is the logic behind this? So far the book examples I've seen use the joint posterior, but this is my first try with conditional posteriors (in non standard form) so I'm not sure I understand the logic.

Question 2 : should I be sampling from my proposal all at once or should I make a decision for each individual $\theta_i$?

Question 3 : should I be using the joint posterior somewhere ? Or is it needed to obtain the conditional posteriors only?

Answer 1

Question 1: Note that $p_2(\theta|y)$ is obtained by integrating out $\alpha$ from $p(\theta, \alpha|y)$. Since you don't care about $\alpha$, this is what you should use - it's the marginal distribution of the vector $\theta | y$ (not the conditional distribution, if by "conditional" you are referring to $\alpha$ rather than $y$.) If you sample from the joint posterior $p(\theta, \alpha|y)$, then ignore $\alpha$, you'll be integrating out $\alpha$ by Monte Carlo instead of by calculus, and, as one might expect, this won't be as accurate.

Question 2: You should sample from your proposal all at once UNLESS you are making separate proposals for each element of $\theta$. If you are making one proposal for $\theta$, the calculations of $q$ will include the effects of the proposals for all the elements of $\theta$, which will break the MCMC sampler's convergence to the posterior if you are actually only going to update one element of $\theta$.

The choice between making proposals for each element of $\theta$ vs. making one joint proposal is really an empirical one that is problem-specific. A block update that has a reasonable average acceptance probability (e.g., 15-40% across the entire run) can be more efficient at covering the region of the parameter space where the posterior lies than separate updates of each element, especially if the elements are correlated and the proposal distribution reflects some or most of that correlation; however, it is often harder to tune the proposal distribution to achieve reasonable acceptance probabilities if it is multivariate rather than a collection of univariate ones.

Question 3: See question 1; it was only needed to obtain the marginal distribution of $\theta|y$ (marginal with respect to $\alpha$, that is.)