Can we change the acceptance rate in random walk Metropolis algorithm by changing the parameter of the proposal distribution?

Question

Can we change the acceptance rate in the random walk Metropolis algorithm by changing the parameter of the proposal distribution?

Let the target distribution be $\pi$. Let $p(x_2 | x_1)$ be the proposal density for a new state $x_2$ at current state $x_1$. The acceptance rate is $$ \alpha = \min(1, \frac{\pi(x_2) p(x_1|x_2)}{\pi(x_1) p(x_2|x_1)}) $$

If I am correct, in the random walk Metropolis algorithm, the proposal density is symmetric in the sense that $p(x_2 | x_1) = p(x_1 | x_2)$, so the acceptance rate doesn't depend on the proposal density, but only on the target distribution $\pi$ to be sampled. So changing the parameter of the proposal distribution will not change the acceptance rate $\alpha$.

For example, if the proposal distribution, at current state $x_1$, is a Gaussian distribution centered at the current state with a constant variance, i.e. $N(x_1, \sigma^2)$, which is by the way symmetric in the above sense, will changing the variance $\sigma^2$ of the Gaussian proposal distribution not change the acceptance rate $\alpha$?

Thanks!

Answer 1

If your proposal has very a low variance, then your proposed new state will be very similar to the current state, so $\frac{\pi(x_2)}{\pi(x_1)}$ will be close to 1 (in the limit, with 0 variance, the proposal and current state will be same and you'll have it exactly equal to 1), so acceptance rate will be close to 100%.

If your proposal has high variance, however, $\frac{\pi(x_2)}{\pi(x_1)}$ will (at least sometimes) be way smaller than 1, so your acceptance rate will get closer and closer to 0%. So the acceptance rate decreases as the variance of the proposal increases.

The problem with very low variances (which will get you higher acceptance rate) is that they take longer to explore the posterior space by never moving far away from the current state. Adaptive MCMC methods like Haario et Al. try to handle such problem by changing the variance matrix of the proposal on the fly.

To tune your acceptance rate you could just try increase and decrease the variance, a somewhat trial and error approach. But depending on the geometry of the posterior, the acceptance rate might change drastically during the sampling process. Also, for multiparameter models the proposal covariance matrix has many variance and covariance terms and such method gets impractical.

There is more sofiscated methods to handle this like the adaptative metropolis method outlined in the link above, or you might want to take a look at other methods like those listed here. You may also try software like Jags and Stan if Metropolis doesn't work for your problem.

Answer 2

I think noting some definitions may be beneficial for future reference to this question and answer.

Ratio of number of accepted proposed states to the number of propositions gives the acceptance rate. Note that acceptance rate is the rate of acceptance over the course of the random walk.

$\alpha$ in the question is called "acceptance probability" by Robert & Casella in their book Introduction to Monte Carlo Methods with R (2010, p. 171). This is very reasonable since $\alpha$, in their presentation, adapted to the notation in the question, is seen here:

$$ x_2= \begin{cases} x_2 & \quad \text{with probability } \alpha(x_2,x_1) \\ x_1 & \quad \text{with probability } 1 - \alpha(x_2,x_1)\\ \end{cases} \\ \\ \text{where }\ \alpha(x_2,x_1) = \min\left\{1, \frac{\pi(x_2)p(x_1 | x_2)}{\pi(x_1)p(x_2 | x_1)}\right\} $$

Now note that $\alpha$ here may becomes independent of the proposal density in case of a random walk proposal when $p(x|y)=p(y|x)$. However, the acceptance rate as defined above is still dependent on it due to the reasons explained by random_user.

Robert and Casella are very clear about differentiating the two and define the latter as "[...] the average of acceptance probability over the iterations."

I have little experience on the matter but it was enough for me to observe that what is referred to in question by "acceptance rate" is sometimes named "acceptance ratio" (see Wikipedia for instance), leading to similar confusions as in the question.