7
$\begingroup$

Model:

The following model corresponds to samples drawn from a Gaussian distribution with unknown mean and unknown variance: \begin{align} x | \mu, \sigma^2 &\sim \mathcal{N}(\mu, \sigma^2 )\\ \mu | \mu_0, \sigma_0^2 &\sim \mathcal{N}(\mu_0, \sigma_0^2)\\ \sigma^2 | \alpha, \beta &\sim Inverse Gamma(\alpha, \beta) \end{align} graghicalmodel

Task:

I want to infer both $\mu$ and $\sigma^2$

Inference:

The conditionals are: \begin{align} p(\mu | \sigma^2, x) &\propto_\mu p(x | \mu, \sigma^2) p(\mu | \mu_0, \sigma_0^2)\\ p(\sigma^2 | \mu) &\propto_{\sigma^2} p(x | \mu, \sigma^2) p(\sigma^2 | \alpha, \beta) \end{align}

Since in the upper equation we have two Normals (which are conjugate with respect to $\mu$), we can easily get the conditional, which is a Normal distribution.

Since in the lower equation we have a normal and an Inverse Gamma (which are conjugate with respect to $\sigma^2$), we can easily get the conditional, which is and Inverse Gamma.

And since we can get the conditional, we can Gibbs sample them both to get their marginal posteriors $p(\mu | x) $ and $p(\sigma^2 | x)$.

Question:

I see that to avoid losing conjugacy $1/\sigma_0$ is forced to be $\rho/\sigma$ (see Michael Jordan's notes or Rasmussen's paper on DP-GMM which I am trying to implement).

But why are we losing conjugacy? Why can't I do this Gibbs sampling using the conditionals above? What am I missing?

EDIT:

I get that the joint posterior of $\mu, \sigma^2$: \begin{align} p(\mu, \sigma^2 | x) \propto p(\mu | \mu_0, \sigma_0) p(\sigma^2 | \alpha, \beta) p (x | \mu, \sigma) \end{align}

cannot be computed from the product of the two conditional posteriors above. But I wonder whether I should use this joint posterior instead of the individual posteriors explained above.

$\endgroup$
  • $\begingroup$ I don't understand your problem. There is semi-conjugacy here. $\endgroup$ – Stéphane Laurent Dec 14 '14 at 14:24
  • $\begingroup$ I'll re-phrase it: with a Gibbs sampler like the above I get $p(\mu | x)$ and $p(\sigma^2 | x)$. What I should need $p(\mu, \sigma^2, x)$ for? (as they do in Rasmussen's paper) $\endgroup$ – alberto Dec 14 '14 at 15:35
3
$\begingroup$

Intuitively, it's because the variance is defined in terms of the mean. So if the experimental mean $\bar x$ is found to be a long way from $\mu_0$, then that increases the posterior estimate of $\sigma^2$.

A bit more rigorously - and I'm taking this from MLAPP 4.6.3.7 - suppose you have a $\text{NI}\chi^2(\mu_0, \kappa_0, \sigma_0, \nu_0)$ prior (which is just a reparameterization of a NIG prior), where $\kappa_0, \nu_0$ encode the strengths of the prior mean and variance, respectively. Then the posterior hyperparameter for $\sigma^2$ is

$$ \sigma^2_N = \frac{1}{\nu_0 + N} \left(\nu_0\sigma^2_0 + Ns^2 + \frac{N\kappa_0}{\kappa_0 + N}(\mu_0 - \bar x)^2\right)$$

where $s^2$ is the experimental variance. We can loosely rewrite this as

$$ \begin{align} \sigma^2_N = \frac{1}{\nu_0 + N} \left(\nu_0 \times \text{contribution from the prior} \\ + N\times \text{contribution from the experiment} \\ + \frac{N\kappa_0}{\kappa_0 + N}\times \text{uncertainty in $\mu$}\right) \end{align}$$

$\endgroup$
  • $\begingroup$ Thanks Andy for that nice answer! However, if I'm not wrong, your answer is about why the mean and the variance are not independent. My question is rather what is the problem with Gibbs sampling each variable conditioned on the other? I edited the question to make it clearer. $\endgroup$ – alberto Dec 14 '14 at 16:04
  • $\begingroup$ Oops, sorry! But that's exactly what the DPGMM paper you linked is addressing, isn't it? You can do the sampling using the two conditionals - what's called the conditionally-conjugate distribution (CCDP) in the paper - or with the full conjugate distribution (CDP). The difference is the CDP gives better performance. $\endgroup$ – Andy Jones Dec 14 '14 at 16:18
  • 2
    $\begingroup$ Yeah, exactly. Ok, I start to see the light... I think Rasmussen does that because he needs the joint posterior in eq. 16b. Therefore there is no problem on sampling both variables separately if we don't need the joint $\mu, \sigma^2$ anywhere. $\endgroup$ – alberto Dec 14 '14 at 16:51
3
$\begingroup$

I do not understand the issue: for the model as described, there is a joint posterior distribution \begin{align} p(\mu, \sigma^2 | x,\mu_0, \sigma_0,\alpha, \beta) \propto \overbrace{p(\mu | \mu_0, \sigma_0)}^\text{Normal} \underbrace{p(\sigma^2 | \alpha, \beta)}_\text{inverse Gamma} \overbrace{p (x | \mu, \sigma^2)}^\text{Normal} \end{align} that is non-standard in the sense that the marginal posterior distributions of both $\mu$ and $\sigma^2$ are not traditional distributions and are not from exponential families. But this distribution on $(\mu, \sigma^2)$ can be computed at any value of $(\mu, \sigma^2,x,\mu_0, \sigma_0,\alpha, \beta)$ and hence can be simulated by all sorts of Monte Carlo approaches, including the or rather a Gibbs sampler.

Therefore, the lack of conjugacy does not prevent one from using a Gibbs sampler or another Markov Chain Monte Carlo method. They both produce simulations from the joint posterior $$p(\mu, \sigma^2 | x,\mu_0, \sigma_0,\alpha, \beta)$$ and not solely from the two marginal posteriors $$p(\mu | x,\mu_0, \sigma_0,\alpha, \beta)\quad\text{and}\quad p(\sigma^2 | x,\mu_0, \sigma_0,\alpha, \beta)$$

$\endgroup$
0
$\begingroup$

Stéphane is right that there's a semi-conjugacy, but you wouldn't lose conjugacy at all if you just expressed the prior and likelihood as distribution having three parameters that work out to be the mean, variance, and number of samples.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.