I was reading this paper http://stat.duke.edu/phd-program/gcc/resources/ResourcesDocuments/CodeExplanation.pdf related to Gibbs sampling.

Suppose we have n iid samples $x_i$ from $N(\mu,\sigma^2)$

Prior probabilities for

$\mu$ ~ $N(m,s^2)$

$\sigma^{-2}$ ~ $Gamma(a,b)$


enter image description here

Can anyone help me with how the conditionals are derived. I tried for $p(\mu|\sigma^2,x)$ which I got something like

$\frac{N(\frac{\sigma^{-2}\mu + s^{-2}m}{\sigma^{-2} + s^{-2}}, \frac{\sigma^2s^2}{\sigma^2+s^2})}{p(x|\sigma^2)} $. Then I didn't get how to proceed and got stuck.

This is what I tried then

$p(\mu|\sigma^2,x) = \frac{N(\frac{\sigma^{-2}\mu + s^{-2}m}{\sigma^{-2} + s^{-2}}, \frac{\sigma^2s^2}{\sigma^2+s^2})}{\int N(\frac{\sigma^{-2}\mu + s^{-2}m}{\sigma^{-2} + s^{-2}}, \frac{\sigma^2s^2}{\sigma^2+s^2})d\mu}$. I don't think this will give me the required expression.


  • $\begingroup$ Some more information about where you're getting stuck would allow a more direct answer. $\endgroup$ – probabilityislogic Jul 27 '13 at 5:35
  • $\begingroup$ @Actually, I got the expression above. I am not sure this is what I am supposed to get. I don't think it will give me the target expression $\endgroup$ – user34790 Jul 27 '13 at 10:29
  • $\begingroup$ @user34790 You may take a look of the book "Bayesian data analysis" (chapter 3) by A. Gelman et al.. $\endgroup$ – semibruin Jul 27 '13 at 22:30

Note that the key to getting the conditionals is to treat all variables except for one as "constant". It is also easier most of the time to work with the log probabilities as these generally add instead of multiply. So for the conditional $(\mu|\sigma^2,x_1,\dots,x_n)$ is written on the log scale as:

$$\log[p(\mu|\sigma^2,x_1,\dots,x_n)]=\log[p(\mu|\sigma^2)]+\log[p(x_1,\dots,x_n|\mu,\sigma^2)]$$ $$-\log[p(x_1,\dots,x_n|\sigma^2)]$$

Now as is usual in this we ignore the last term, as it is not a function of $\mu$ (or put another way, we can implicitly calculate it by normalising the probability distribution so that it integrates to 1). Also, we can write the log-likelihood for all the data as a function of the individual log-likelihoods. So we have for the prior:

$$\log[p(\mu|\sigma^2)]=\log[p(\mu)]=-\frac{1}{2}\left(\log(2\pi s^2)+\frac{(\mu-m)^2}{s^2}\right)$$ $$=c_1-\frac{1}{2s^2}\mu^2+\frac{m}{s^2}\mu$$

Where $c_1$ is a number which depends only on $m$ and $s^2$ but not on $\mu$. Thi part of a probability distribution is often known as the "kernel", although kernels usually aren't written on the log scale. Note that identifying the kernel is equivalent to identifying the probability distribution. The log-likelihood has a similar form:

$$\log[p(x_1,\dots,x_n|\mu,\sigma^2)]=\sum_{i=1}^{n}\log[p(x_i|\mu,\sigma^2)]$$ $$=\sum_{i=1}^{n}-\frac{1}{2}\left(\log(2\pi\sigma^2)+\frac{(\mu-x_i)^2}{\sigma^2}\right)$$ $$=-\frac{1}{2}\sum_{i=1}^{n}\left(\log(2\pi\sigma^2)+\frac{x_i^2}{\sigma^2}+\frac{\mu^2-2\mu x_i}{\sigma^2}\right)$$ $$=c_2-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(\mu^2-2\mu x_i)=c_2-\frac{n}{2\sigma^2}\mu^2 +\frac{n\overline{x}}{\sigma^2}\mu$$

Where $c_2$ depends on the data $x_1,\dots,x_n$ and on $\sigma^2$, but not on $\mu$. Combining the likelihood and the prior gives you:

$$\log[p(\mu|\sigma^2,x_1,\dots,x_n)]=c_1-\frac{1}{2s^2}(\mu^2-2m\mu)+c_2-\frac{n}{2\sigma^2}(\mu^2-2\mu \overline{x})$$ $$-\log[p(x_1,\dots,x_n|\sigma^2)]$$ $$=c_0-\frac{1}{2}\left(\frac{1}{s^2}+\frac{n}{\sigma^2}\right)\mu^2+\left(\frac{m}{s^2}+\frac{n\overline{x}}{\sigma^2}\right)\mu $$

Where $c_0=c_1+c_2-\log[p(x_1,\dots,x_n|\sigma^2)]$

This is the same form as the prior, but with different values for $m$ and $s^2$. So the posterior is a normal distribution. On comparing the two equations, we see that the mean and variance of the posterior satisfy:

$$-\frac{1}{2s_*^2}=-\frac{1}{2}\left(\frac{1}{s^2}+\frac{n}{\sigma^2}\right)$$ $$\frac{m^*}{s_*^2}=\frac{m}{s^2}+\frac{n\overline{x}}{\sigma^2}$$

Solving these equations gives you the mean and variance. I hope this approach will help you get the conditional for $\sigma^2$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.