# Computational aspect of the Metropolis-Hastings algorithm

One of the examples online is about how to write the Metropolis-Hastings algorithm from scratch. This tutorial uses a linear regression model as an example. Estimate three parameters: an intercept, a slope, and a standard deviation.

The code in that tutorial

1. proposes three candidate samples (theta_star) from proposal distribution (normal distribution),
2. calculates posterior (a number): sum(loglikelihoods) + b0_prior + b1_prior + sd_prior (all on log scale),
3. calculates the acceptance probability (a number) and compares it with a random number.
4. accepts all the three elements or rejects them all.

Can I use the alternative way below? Say, if I calculate the posterior respectively like

sum(loglikelihoods) + b0_prior
sum(loglikelihoods) + b1_prior
sum(loglikelihoods) + sd_prior


where the three sum(loglikelihoods) above are same in value. Only calculated once, not re-calculated three times.

Then I get the posterior (3 by 1 vector), calculate acceptance probability (3 by 1 vector), and compare them with three random numbers. So the three elements are updated respectively. Not accept all or reject all.

Is this modified procedure still a valid Metropolis-Hastings algorithm? I feel my question is related with this post, but not sure the above is correct in principle and theory.

• A second comment is that changing one component of the parameter $\theta$ at a time, $\beta_0$ then $\beta_1$ then $\sigma$ is called Gibbs sampling. Feb 17 at 13:17
• Sorry, they are typos. Allow me to edit the question again. Feb 17 at 13:18
• The general description of Metropolis-Hastings in the reference is confusing, the acceptance ratio should involve the proposal density. I suggest you instead read a genuine textbook on the topic. Feb 17 at 13:24
• Thank you, Professor. I just used that post as example to ask my question. As your second comment means, my procedure is simply Gibbs sampling? It lets me want to ask: Gibbs sampling updates element by element by conditional distributions. That is, update theta at a time, then beta_0, then sigma. This procedure needs to calculate the loglikelihood three times, right? In my question, I only calculate loglikelihood once and update the three elements in parallel. It seems not the same. I wonder if the modified procedure in my question is correct or not. Thank you for your kind reply above. Feb 17 at 13:35
• Yes you need to update the likelihood three times. Otherwise it is not correct. Feb 17 at 15:47

Above is an illustration of the difference between both approaches, when applied to a toy target distribution

a<-function(b){
dnorm(b[1],l=TRUE)+
dnorm(b[2]-.5*b[1]^2,sd=.7,l=TRUE)+
dnorm(b[3]+1.2*b[2]+.5*b[1]^2,sd=.5,l=TRUE)}


The original (and correct) Metropolis-within-Gibbs scheme

g<-function(T=1e4){
o=matrix(rnorm(3),T,3)
for(t in 2:T){
p=o[t-1,]+rnorm(3)
o[t,]=p+(log(runif(1))>a(p)-a(o[t-1,]))*(o[t-1,]-p)}
return(o)}


produces golden dots/samples. The one-choice at a time alternative

k<-function(T=1e4){
o=matrix(rnorm(3),T,3)
for(t in 2:T){
p=o[t-1,]+rnorm(3)
b=a(p);d=a(o[t-1,])
for(i in 1:3) o[t,i]=p[i]+(log(runif(1))>b-d)*(o[t-1,i]-p[i])}
return(o)}


produces red dots/samples. (Upper row is comparing samples two coordinates at a time, lower row is comparing density estimates one coordinate at a time.)

• Really thank you for the time to provide a toy example! Does it mean the one-choice-at-a-time alternative (incorrect method) cannot produce an approximate distribution to the true distribution? If so, it would be clear to impose true distribution on the density plots. Feb 21 at 15:35