Calculating acceptance probability in Metropolis-Hastings algorithm In the Metropolis-Hastings algorithm, acceptance probability is given as
$$
\alpha = \min \left( 1,\frac{f(\theta^{'}|y)q(\theta|\theta^{'})}{f(\theta|y)q(\theta^{'}|\theta)} \right)
$$
which simplifies to
$$
\alpha = \min \left( 1,\frac{f(y|\theta^{'})f(\theta^{'}) q(\theta|\theta^{'})}{f(y|\theta)f(\theta) q(\theta^{'}|\theta)} \right)
$$
My questions are:

*

*Is it possible to evaluate $f(y|\theta^{'})$ and $f(y|\theta)$ without knowing its analytical form?

*If not, does that mean that we need to know the analytical form of the posterior and the likelihood to use the Metropolis-Hastings algorithm? Also, what is the purpose of the data then?

*If yes, how do we do it?

Thank you in advance!
 A: *

*It is technically possible, but usually unnecessary. It is usually far easier to calculate the likelihood and the prior than it is the posterior.


*No, you do not need to know the posterior a priori. The reason we use Metropolis Hastings is because the full expression for the posterior has a normalizing constant that is difficult to calculate. BUT, you do know it is a constant, that it will make the probabilities sum to 1, and that the posterior is proportional to the likelihood times the prior. If you sample the distribution proportional to the likelihood times the prior, then you can just divide by the total number of samples to get the posterior density instead. The data's purpose is as a test: if the data fit a particular model better, that model will be more likely to be correct.


*Look up "Bayesian Statistical Likelihood", which essentially works by simulating the possible experimental outcomes based on the current model, and constructing a likelihood function around the simulations. That said, this is usually overkill.
A: When
$$\alpha = \min \left( 1,\frac{f(y|\theta^{'})f(\theta^{'}) q(\theta|\theta^{'})}{f(y|\theta)f(\theta) q(\theta^{'}|\theta)} \right)$$
involves an intractable likelihood function $f(y|\cdot)$ that cannot be computed, several (exact) alternatives are available:

*

*the intractable part of $f(y|\theta)$ may also appear in $q(\theta|\theta')$ and hence cancels in the ratio. This is the idea of the auxiliary variable device of Møller et al. (2006). Also pursued by Murray et al.  (2012). They mostly address the setup of doubly intractable distributions where the likelihood function $f(y|\theta)$ involves a multiplicative factor $\mathfrak c(\theta)$ that is itself intractable.


*the intractable likelihood $f(y|\theta)$ may unbiasedly estimated by a random variable $\xi(y,\theta)$, even up to a normalising constant: $$\mathbb E[\xi(y,\theta)]=\alpha(y)f(y|\theta)$$ where $\alpha(y)$ may be unknown / intractable. This is the idea of pseudo-marginal MCMC of Andrieu & Roberts (2009).


*Demarginalising $y$ into $(y,z)$ and $f(y|\theta)$ into $\tilde f(y,z|\theta)$ such that $$\int_{\mathbb Z} \tilde f(y,z|\theta)\,\text dz=f(y|\theta)$$and $\tilde f(y,z|\theta)$ tractable is a more general auxiliary variable method, where the augmented $(\theta,z)$ is simulated conditional on $y$ through an MCMC method. When using a Gibbs sampler, the ratio $\alpha$ may then be replaced at iteration $t$ by
$$\tilde\alpha = \min \left( 1,\frac{\tilde f(y,z^t|\theta^{'})f(\theta^{'}) q(\theta^t|\theta^{'})}{f(y,z^t|\theta^t)f(\theta^t) q(\theta^{'}|\theta^t)} \right)$$
equivalent to
$$\tilde\alpha = \min \left( 1,\frac{\tilde f(y|\theta^{'},z^t)f(\theta^{'}) q(\theta^t|\theta^{'})}{f(y|\theta^t,z^t)f(\theta^t) q(\theta^{'}|\theta^t)} \right)$$
which is a special case of 1.
If none of these (related) approaches can be used (in a sufficiently efficient manner), then a approximate approach is to resort to ABC (Approximate Bayesian computation).
