3
$\begingroup$

Suppose there is a one-dimensional normal distribution $\mathcal{N}(\mu, \sigma)$ for which we want to infer the joint distribution of the parameters using Gibbs sampling. Let $D$ be the data, consisting of $n$ datapoints $d_1, ..., d_n$. Assume a broad Gamma prior to the precision parameter $\beta=\frac{1}{\sigma^2}$.

So, now I need to find an expression for $P(\sigma | D, \mu)$. Note that $P(\sigma | D, \mu) \sim P(D| \mu, \sigma) P(\sigma)$.

I tried the following:

\begin{align} P(\sigma | D, \mu) &\propto P(D| \mu, \sigma) P(\sigma) \\ &\propto \Gamma(\sigma | \alpha, \beta) \prod\limits_{i=1}^n \mathcal{N}(d_i| \mu, \sigma) \end{align}

In the next step, I throw away the $(\sqrt{2 \pi})^n$ since that is a constant.

\begin{align} \qquad\qquad\qquad\qquad\qquad\qquad &\propto_\sigma \Gamma(\sigma | \alpha, \beta) \sigma^{-n} \prod\limits_{i=1}^n \exp(-\frac{\sum\limits_{i=1}^n (d_i - \mu)^2}{2 \sigma^2}) \\ &\propto \Gamma(\sigma | \alpha, \beta) \sigma^{-n} \exp(-\frac{\sum\limits_{i=1}^n (d_i - \mu)^2}{2 \sigma^2}) \\ &\propto \beta^{\alpha} \sigma^{\alpha-1} \exp(-\beta \sigma) \sigma^{-n} \exp(-\frac{\sum\limits_{i=1}^n (d_i - \mu)^2}{2 \sigma^2}) \\ &\propto \sigma^{\alpha-1-n} \exp(-\frac{\sum\limits_{i=1}^n (d_i - \mu)^2 - 2 \beta \sigma^3}{2 \sigma^2}) \end{align}

And now? Is it supposed to be another Gamma distribution? What steps am I missing? I don't see it yet.

$\endgroup$
  • $\begingroup$ Your prior on $\sigma$ or $\sigma^{-1}$ cannot involve $\sigma$ as a parameter. If you assume a $\text{G}(a,b)$ prior on $\sigma^{-2}$, the parameters $a$ and $b$ must be fixed. $\endgroup$ – Xi'an Oct 18 '16 at 18:39
  • $\begingroup$ Thank you, I tried to change it, hopefully the partial answer is correct now. $\endgroup$ – www.data-blogger.com Oct 18 '16 at 18:47
  • 1
    $\begingroup$ Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. $\endgroup$ – gung Oct 18 '16 at 18:56
  • 1
    $\begingroup$ The Gamma prior should be on $\sigma^{-2}$ to enjoy conjugacy. $\endgroup$ – Xi'an Oct 18 '16 at 19:39
0
$\begingroup$

Is it supposed to be another Gamma distribution?

Yes, and that is not surprising. It is related to the concept of 'conjugate priors'. When using conjugate priors, the posterior and prior distributions of a parameter belong to the same family.

I am not sure about your equations, but if you are just looking for the answer, you can find it in this Wikipedia page: https://en.wikipedia.org/wiki/Conjugate_prior .

In the section 'Table of conjugate distributions', in 'Continuous distributions' table, look at 'Normal with known mean'. That is exactly what you need in this Gibbs sampling problem, since it gives you a solution to $P(\sigma | D, \mu)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.