I am reading a book on doing Bayesian Data Analysis. I have just learned what the Metropolis Hastings (MH) Algo does, at least in relation to Bayesian Data Analysis. My understanding of the MH Algo is that it allows us to approximate the posterior distribution for which analytically it may be tough for us to determine. The biggest issue seems to be evaluating the denominator in Bayes' theorem as for continuous priors, we need to evaluate an integral to normalize the likelihood function.

In the book I'm reading, I just got to a section on Gibbs Sampling. I am having a hard time understanding how it works and what the point of it is. In the book, it says that "Gibbs Sampling is a special version of the Metropolis-Hastings Algorithm". Here are the things I don't understand.

  • When running the algo for Gibbs Sampling, the next point we travel to is determined via a conditional distribution. The book defines this distribution (I am assuming it is the proposal distribution) as p(θ1|{θj-1},D), where D is the data or real world sample you have observed. The above distribution looks just like the posterior distribution as it's basically a parameter value given the Data. If the goal of MH MCMC was to generate the posterior distribution because analytically it is too hard, how do we already have the posterior distribution to use for Gibbs Sampling? And what's the point of sampling from the posterior distribution anyways if we already know what is. We should be able to extract the density at any point just by using the formula for the posterior distribution.
  • My second question is that, in Gibbs Sampling, the book states we move from one point to another with 100% probability. I don't understand this as isn't the mechanic that makes MH MCMC work is the probability of moving to one point or another depends on the relative likelihood. The ratio of likelihoods between two points -> probability of moving to that point -> allows us to approximate the posterior distribution.


  • $\begingroup$ 1. See Mathjax tutorial and quick reference to set your mathematics to be readable (particularly in relation to subscripts/superscripts) 2. Please define your symbols - what's θ1 and θj-1 ? 3. What would happen in MH if the proposal distribution was the thing you wanted to sample from? $\endgroup$
    – Glen_b
    Oct 28, 2019 at 22:53
  • $\begingroup$ Whoever linked me the other post, that's exactly what I was looking for. He's actually reading the exact same book as me and stuck on the exact same page. Maybe the author isn't clear if multiple people are confused about that same thing ha! $\endgroup$
    – confused
    Oct 29, 2019 at 15:00