My question concerns Gibbs sampling. Suppose that I have three unknown quantities, $\mu, \sigma^2$ and $c$. I have given prior information and I have given the likelihood which allows me to compute the posterior $g(\mu, \sigma^2, c \ | \ \mbox{data})$. Now, I want to write a Gibbs sampler to generate from the posterior. To that end, the exercise says to consider the conditional distributions $[c \ | \ \mu, \sigma^2]$ and $[\mu, \sigma^2 \ | \ c]$.

Using the posterior I can easily find the corresponding distribution for $[c | \ \mu, \sigma^2]$. However, the conditional density $[\mu, \sigma^2 \ | \ c]$ does not have a known functional form. Therefore, my initial idea was to split it as follows: $$g[\mu, \sigma^2 \ | \ c] = g(\mu \ | \ \sigma^2,c) \underbrace{g(\sigma^2 \ | c)}_{(A)},$$ where $(A)$ can be found by integrating $g(\mu, \sigma^2 \ | c)$ w.r.t $\mu$. This gives me some nice distributions to work with, however, I do not have a dependence with $\mu$ anymore in $(A)$ since it has been integrated out. Therefore, I don't think this is the correct way. Instead, to write the Gibbs sampler I should consider the following conditional densities:

  1. $g(c \ | \ \mu, \sigma^2)$
  2. $g(\mu \ | \ c, \sigma^2)$
  3. $g(\sigma^2 \ | \mu, c)$.

Question: is the first idea wrong? I guess we can use it but it is not a Gibbs sampler then?


The exercise asked you implement a Gibbs sampler using the full conditionals of $[c \mid \mu, \sigma^2]$ and $[\mu, \sigma^2 \mid c]$. This is a valid Gibbs sampler and has the special name of Blocked Gibbs Sampler, since $\mu$ and $\sigma^2$ are treated as one variable using their joint conditional distribution. To see this think of $\theta = (\mu, \sigma^2)$. Then the blocked Gibbs sampler would look a usual Gibbs sampler with the conditionals being $[c \mid \theta]$ and $[\theta \mid c]$. The tricky part is, how do you find $[\theta \mid c]$? Your idea of using $$g[\theta \mid c] = g[\mu, \sigma^2 \mid c] = g[\mu \mid \sigma^2, c]\, \underbrace{g[\sigma^2 \mid c]}_{(A)}$$

is absolutely the right idea and is correct. With these distributions, your blocked Gibbs sampler will update as follows. Let $c_0, \mu_0$ and $\sigma^2_0$ will be the starting values.

  1. $c_1 \sim g[c \mid \mu_0, \sigma^2_0]$
  2. $\sigma^2_1 \sim g[\sigma^2 \mid c_1]$
  3. $\mu_1 \sim g[\mu \mid \sigma^2_1, c_1]$

and so on.

  • $\begingroup$ Thank you for the answer. Suppose that the exercise has asked to implement a Gibbs sampler without explicitly telling me to use $[c\ | \ \mu, \sigma]$ and $[\mu, \sigma^2 \ | \ c]$ then the second approach would be the typical way to go, right? $\endgroup$ – Cavents Jun 10 '16 at 16:26
  • 1
    $\begingroup$ Yes, then the second approach would be the typical way. There is sometimes benefit to using the blocked Gibbs sampler, as it has been shown that it can be more efficient/have better properties. However, it is generally unclear if one is better than the other. $\endgroup$ – Greenparker Jun 10 '16 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.