Compute the likelihood in Metropolis–Hastings: How does it relate to a posterior in Bayesian Analysis?

Question

Basic question about MCMC Metropolis–Hastings algorithm. I am trying to understand the Metropolis–Hastings algorithm and it's connection to Bayesian Analysis. Suppose I want to construct an MCMC MH algrorithm to evaluate my posterior distribution in the my Bayesian Analysis.

I am looking at the step where $\alpha$ is being computed: $$\alpha = \frac{P(\theta^* | \textbf{Y})}{P(\theta^{(i)} | \textbf{Y})}$$ Here I have (for simplicity) assumed that my prposal density in MH algorithm is symmetric.

$P(\theta^* | \textbf{Y})$ is the likelihood of $\theta^*$ given our data. So basically before the MH algorithm we need to assume a distribution for $\theta$, right? And is $P(\theta^* | \textbf{Y})$ our posterior distribution? How do I compute $P(\theta^* | \textbf{Y})$?

Also, feel free to provide an oral (in words) explanation of $P(\theta^* | \textbf{Y})$ and it's connection the posterior distribution we wish to compute. My confusion arises with the fact that we are not able to know our posterior distribution and that is the whole point of MH. So how can comput the likelihood ...

Answer 1

You have still a lot of confusion. I'll try to explain with an example.

Let suppose $\mathbf{Y}= (y_1, \dots ,y_n)'$,and you have $$ Y_i = \beta_0+\beta_1x_i+\epsilon_i $$, with $$ \epsilon \sim N(0,\sigma^2) $$ i.e. a regression model. Then your parameters are $\theta= (\beta_0, \beta_1, \sigma^2)'$. The observation, given $\theta$, are independent and then the likelihood is $$ f(\mathbf{Y}|\theta)= \prod_{i=1}^n f(y_i|\theta) $$ where $f(y_i|\theta)$ is normal with mean $\beta_0+\beta_1x_i$ and variance $\sigma^2$.

As you can see, you can generally compute $f(\mathbf{Y}|\theta)$ since it is your model, it is what you assume.

Of course you have also to define prior for $\beta_0$, $\beta_1$ and $\sigma^2$, that is $f(\theta)= f(\beta_0)f(\beta_1)f(\sigma^2)$, for example.

$f(\mathbf{Y}|\theta)$ is not equal to $f(\mathbf{Y})$ because the normal likelihood depends on the parameters $theta$.

Now, if you want to have posterior samples, $f(\theta|Y)$, you start defining some initial value $\beta_0^0$, $\beta_1^0$ and $\sigma^{2,0}$, you propose some new values, $\beta_0^*$, $\beta_1^*$ and $\sigma^{2,*}$, and using the answer that I gave previously you can update your parameters. Notice that $$ f(\mathbf{Y}|\theta^*)= \prod_{i=1}^n f(y_i|\theta^*) $$ where $f(y_i|\theta^*)$ is normal with mean $\beta_0^*+\beta_1^*x_i$ and variance $\sigma^{2,*}$.

BTW: you don't need to compute the posterior, the idea is to get samples from the psoterior $f(\theta|\mathbf{Y})$

Answer 2

There is a lot of confusion. You want to evaluate the posterior $$ f(\theta|\mathbf{y}) =\frac{f(\mathbf{y}|\theta)f(\theta)}{f(\mathbf{y})} $$ where I use $f()$ to indicate a density, to be as general as possible. You have to decide the likelihood of your model, i.e., $f(\mathbf{y}|\theta)$, and the prior over $\theta$, i.e., $f(\theta)$. In other words, you have to assume a particular distribution for each of them. Then you know the functional forms of both.

To obtain samples from the posterior using an MH, you choose a starting value of $\theta$, that we call $\theta^0$. You now propose a value, from some symmetrical distribution (to make things as simple as possible), indicated with $\theta^*$. We then compute the following $$ \alpha = \min(1,\frac{f(\mathbf{y}|\theta^*)f(\theta^*)}{f(\mathbf{y}|\theta^0)f(\theta^0)}) $$ and we sample from the uniform distribution $$ u \sim \text{Unif}(0,1). $$ Now, if $u<\alpha$ you set $\theta^1=\theta^*$, otherwise $\theta^1=\theta^0$.

Then you repeat the same steps to find the values of $\theta^2,\theta^3,...$, until convergence.