Gibbs sampling, what to use?

Question

My question concerns Gibbs sampling. Suppose that I have three unknown quantities, $\mu, \sigma^2$ and $c$. I have given prior information and I have given the likelihood which allows me to compute the posterior $g(\mu, \sigma^2, c \ | \ \mbox{data})$. Now, I want to write a Gibbs sampler to generate from the posterior. To that end, the exercise says to consider the conditional distributions $[c \ | \ \mu, \sigma^2]$ and $[\mu, \sigma^2 \ | \ c]$.

Using the posterior I can easily find the corresponding distribution for $[c | \ \mu, \sigma^2]$. However, the conditional density $[\mu, \sigma^2 \ | \ c]$ does not have a known functional form. Therefore, my initial idea was to split it as follows: $$g[\mu, \sigma^2 \ | \ c] = g(\mu \ | \ \sigma^2,c) \underbrace{g(\sigma^2 \ | c)}_{(A)},$$ where $(A)$ can be found by integrating $g(\mu, \sigma^2 \ | c)$ w.r.t $\mu$. This gives me some nice distributions to work with, however, I do not have a dependence with $\mu$ anymore in $(A)$ since it has been integrated out. Therefore, I don't think this is the correct way. Instead, to write the Gibbs sampler I should consider the following conditional densities:

1. $g(c \ | \ \mu, \sigma^2)$
2. $g(\mu \ | \ c, \sigma^2)$
3. $g(\sigma^2 \ | \mu, c)$.

Question: is the first idea wrong? I guess we can use it but it is not a Gibbs sampler then?

Answer 1

The exercise asked you implement a Gibbs sampler using the full conditionals of $[c \mid \mu, \sigma^2]$ and $[\mu, \sigma^2 \mid c]$. This is a valid Gibbs sampler and has the special name of Blocked Gibbs Sampler, since $\mu$ and $\sigma^2$ are treated as one variable using their joint conditional distribution. To see this think of $\theta = (\mu, \sigma^2)$. Then the blocked Gibbs sampler would look a usual Gibbs sampler with the conditionals being $[c \mid \theta]$ and $[\theta \mid c]$. The tricky part is, how do you find $[\theta \mid c]$? Your idea of using $$g[\theta \mid c] = g[\mu, \sigma^2 \mid c] = g[\mu \mid \sigma^2, c]\, \underbrace{g[\sigma^2 \mid c]}_{(A)}$$

is absolutely the right idea and is correct. With these distributions, your blocked Gibbs sampler will update as follows. Let $c_0, \mu_0$ and $\sigma^2_0$ will be the starting values.

1. $c_1 \sim g[c \mid \mu_0, \sigma^2_0]$
2. $\sigma^2_1 \sim g[\sigma^2 \mid c_1]$
3. $\mu_1 \sim g[\mu \mid \sigma^2_1, c_1]$

and so on.