Use of Metropolis-Hastings in Bayesian Inference

Question

I am now studying the Metropolis-Hastings algorithm and I want to apply it in order to made a Bayesian Inference of a function $y=f(x)$ to a dataset $D=\{x_i,y_i\}$. Five parameters of the function need to be determined $\theta=\{\theta_1,...,\theta_5\}$.

Here I want to first introduce how to develop the M-H algorithm.

I start from an initial guess of the model parameters $\theta^{(0)}$ and, using a candidate distribution (in my case a Multivariate Normal Distribution centred in $\theta^{(0)}$ with a certain variance $\delta$ that has to be tuned), select a candidate $\theta^{(*)}$. Since the candidate distribution is symmetrical, the acceptance probability is the ratio:

\begin{equation} \alpha=min\{1, \frac{\pi(\theta^{(*)})}{\pi(\theta^{(0)})} \} \end{equation} Where $\pi(\bullet)$ is the target function, proportional to the Posterior Distribution.

$\theta^{(*)}$ is accepted ($\theta^{(1)}=\theta^{(*)}$) if $\alpha$ is bigger than $u=U(0,1)$, otherwise it is rejected ($\theta^{(1)}=\theta^{(0)}$). A new candidate can be drawn and the process repeated.

In case I want to use the M-H algorithm in a inference problem, the target function $\pi(\theta)$ can be substituted by the Likelihood function $L(\theta)$.

However, if I want to made a Bayesian inference, it is not really clear to me the role of the prior distributions associated to the elements $\theta_i$ of the vector parameter. How do they figure in the acceptance probability?

Thanks for the answers and the comments

Answer 1

The posterior distribution can be expressed as $$ p(\theta|y) = \frac{p(y|\theta)\,p(\theta)}{p(y)} $$ where $$ p(y) = \int p(y|\theta)\,p(\theta)\,d\theta. $$ The likelihood is given by $L(\theta) := p(y|\theta)$ and the prior is $p(\theta)$. The target function can be expressed as $$ \pi(\theta) = c\,L(\theta)\,p(\theta), $$ where $c^{-1} = p(y)$. The (possibly unknown) constant of proportionality $c$ cancels out in the ratio. The algorithm then proceeds as outlined.