0
$\begingroup$

One of the examples online is about how to write the Metropolis-Hastings algorithm from scratch. This tutorial uses a linear regression model as an example. Estimate three parameters: an intercept, a slope, and a standard deviation.

The code in that tutorial

  1. proposes three candidate samples (theta_star) from proposal distribution (normal distribution),
  2. calculates posterior (a number): sum(loglikelihoods) + b0_prior + b1_prior + sd_prior (all on log scale),
  3. calculates the acceptance probability (a number) and compares it with a random number.
  4. accepts all the three elements or rejects them all.

Can I use the alternative way below? Say, if I calculate the posterior respectively like

`sum(loglikelihoods)` + `b0_prior`
`sum(loglikelihoods)` + `b1_prior`
`sum(loglikelihoods)` + `sd_prior`

where the three sum(loglikelihoods) above are same in value. Only calculated once, not re-calculated three times.

Then I get the posterior (3 by 1 vector), calculate acceptance probability (3 by 1 vector), and compare them with three random numbers. So the three elements are updated respectively. Not accept all or reject all.

Is this modified procedure still a valid Metropolis-Hastings algorithm? I feel my question is related with this post, but not sure the above is correct in principle and theory.

Improve this question
$\endgroup$
5
  • $\begingroup$ A second comment is that changing one component of the parameter $\theta$ at a time, $\beta_0$ then $\beta_1$ then $\sigma$ is called Gibbs sampling. $\endgroup$
    – Xi'an
    Feb 17 at 13:17
  • $\begingroup$ Sorry, they are typos. Allow me to edit the question again. $\endgroup$
    – Keith Lau
    Feb 17 at 13:18
  • $\begingroup$ The general description of Metropolis-Hastings in the reference is confusing, the acceptance ratio should involve the proposal density. I suggest you instead read a genuine textbook on the topic. $\endgroup$
    – Xi'an
    Feb 17 at 13:24
  • $\begingroup$ Thank you, Professor. I just used that post as example to ask my question. As your second comment means, my procedure is simply Gibbs sampling? It lets me want to ask: Gibbs sampling updates element by element by conditional distributions. That is, update theta at a time, then beta_0, then sigma. This procedure needs to calculate the loglikelihood three times, right? In my question, I only calculate loglikelihood once and update the three elements in parallel. It seems not the same. I wonder if the modified procedure in my question is correct or not. Thank you for your kind reply above. $\endgroup$
    – Keith Lau
    Feb 17 at 13:35
  • 1
    $\begingroup$ Yes you need to update the likelihood three times. Otherwise it is not correct. $\endgroup$
    – Xi'an
    Feb 17 at 15:47

1 Answer 1

Reset to default
0
$\begingroup$

enter image description here Above is an illustration of the difference between both approaches, when applied to a toy target distribution

a<-function(b){
dnorm(b[1],l=TRUE)+
dnorm(b[2]-.5*b[1]^2,sd=.7,l=TRUE)+
dnorm(b[3]+1.2*b[2]+.5*b[1]^2,sd=.5,l=TRUE)}

The original (and correct) Metropolis-within-Gibbs scheme

g<-function(T=1e4){
  o=matrix(rnorm(3),T,3)
  for(t in 2:T){
    p=o[t-1,]+rnorm(3)
    o[t,]=p+(log(runif(1))>a(p)-a(o[t-1,]))*(o[t-1,]-p)}
  return(o)}

produces golden dots/samples. The one-choice at a time alternative

k<-function(T=1e4){
   o=matrix(rnorm(3),T,3)
   for(t in 2:T){
      p=o[t-1,]+rnorm(3)
      b=a(p);d=a(o[t-1,])
      for(i in 1:3) o[t,i]=p[i]+(log(runif(1))>b-d)*(o[t-1,i]-p[i])}
   return(o)}

produces red dots/samples. (Upper row is comparing samples two coordinates at a time, lower row is comparing density estimates one coordinate at a time.)

Improve this answer
$\endgroup$
1
  • $\begingroup$ Really thank you for the time to provide a toy example! Does it mean the one-choice-at-a-time alternative (incorrect method) cannot produce an approximate distribution to the true distribution? If so, it would be clear to impose true distribution on the density plots. $\endgroup$
    – Keith Lau
    Feb 21 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.