Do I need to evaluate acceptance rates in Metropolis within Gibbs algorithm?

Question

Consider the Gibbs sampler

1. Sample $\theta' \sim p(\theta|\tau, D)$
2. Sample $\tau' \sim p(\tau|\theta', D)$

where $\theta,\tau$ parameters of the data $D$. Now assume that we can only sample from $p(\tau|\theta', D)$ using a Metropolis step, for example because its normalization constant is unknown but we can evaluate the unnormalized density. In the Metropolis algorithm a proposal sample is drawn from a proposal distribution, such as the normal distribution. The step size is in this case determined by the variance of the normal, which is a tuning parameter for the acceptance rate of the Metropolis sampler.

At a given step in the Gibbs sampler, we could sample a large number of times from $p(\tau|\theta', D)$ i.e. holding $\theta'$ constant across repeated draws. Then we could estimate the acceptance rate by checking which proportion of proposals was accepted. However in the Metropolis-within-Gibbs sampler suggested above it is not common to draw a large number of samples from $p(\tau|\theta', D)$ at a given step; instead one sample is drawn by Metropolis and then the Gibbs sampler continues at the next iteration. Suppose that this next iteration yields $\theta''$ (two dashes), then the next Metropolis step in the second iteration will sample from $p(\tau|\theta'', D)$.

Now my specific question is: the optimal tuning parameters for the proposal distributions used for sampling from $p(\tau|\theta', D)$ at the first and $p(\tau|\theta'', D)$ at the second iteration may indeed be different. Assuming I can only set the tuning parameter once, how should I monitor acceptance rates in this Metropolis-Gibbs sampler? Is it enough to do this once across draws of $\tau$?

Answer 1

Usually, you set the tuning parameter once and evaluate the overall acceptance probability ex post once over all iterations of the sampler. Here, you aim for the ''golden acceptance ratio'' of 23,4%. In case your acceptance ratio is higher, this translates into the proposal variance being to small, leading to too many accepts as the posterior distribution is explored very slowly. In case your acceptance ratio is too low, the contrary holds and you can lower the proposal variance. However, that might be a cumbersome process as it requires multiple runs of the chain and manual tuning.

There are alternatives out there such as adaptive MH steps that automatically set the tuning parameters, which might facilitate analysis depending on the specific situation.

I've personally never heard of taking a lot of draws in each iteration of the Gibbs sampler and evaluate the acceptance ratio for each draw.