Why do we lose conjugacy when assuming unknown $\mu$ and unknown $\sigma^2$ in a normal distribution?

Question

Model:

The following model corresponds to samples drawn from a Gaussian distribution with unknown mean and unknown variance: \begin{align} x | \mu, \sigma^2 &\sim \mathcal{N}(\mu, \sigma^2 )\\ \mu | \mu_0, \sigma_0^2 &\sim \mathcal{N}(\mu_0, \sigma_0^2)\\ \sigma^2 | \alpha, \beta &\sim Inverse Gamma(\alpha, \beta) \end{align}

Task:

I want to infer both $\mu$ and $\sigma^2$

Inference:

The conditionals are: \begin{align} p(\mu | \sigma^2, x) &\propto_\mu p(x | \mu, \sigma^2) p(\mu | \mu_0, \sigma_0^2)\\ p(\sigma^2 | \mu) &\propto_{\sigma^2} p(x | \mu, \sigma^2) p(\sigma^2 | \alpha, \beta) \end{align}

Since in the upper equation we have two Normals (which are conjugate with respect to $\mu$), we can easily get the conditional, which is a Normal distribution.

Since in the lower equation we have a normal and an Inverse Gamma (which are conjugate with respect to $\sigma^2$), we can easily get the conditional, which is and Inverse Gamma.

And since we can get the conditional, we can Gibbs sample them both to get their marginal posteriors $p(\mu | x) $ and $p(\sigma^2 | x)$.

Question:

I see that to avoid losing conjugacy $1/\sigma_0$ is forced to be $\rho/\sigma$ (see Michael Jordan's notes or Rasmussen's paper on DP-GMM which I am trying to implement).

But why are we losing conjugacy? Why can't I do this Gibbs sampling using the conditionals above? What am I missing?

EDIT:

I get that the joint posterior of $\mu, \sigma^2$: \begin{align} p(\mu, \sigma^2 | x) \propto p(\mu | \mu_0, \sigma_0) p(\sigma^2 | \alpha, \beta) p (x | \mu, \sigma) \end{align}

cannot be computed from the product of the two conditional posteriors above. But I wonder whether I should use this joint posterior instead of the individual posteriors explained above.

Answer 1

Intuitively, it's because the variance is defined in terms of the mean. So if the experimental mean $\bar x$ is found to be a long way from $\mu_0$, then that increases the posterior estimate of $\sigma^2$.

A bit more rigorously - and I'm taking this from MLAPP 4.6.3.7 - suppose you have a $\text{NI}\chi^2(\mu_0, \kappa_0, \sigma_0, \nu_0)$ prior (which is just a reparameterization of a NIG prior), where $\kappa_0, \nu_0$ encode the strengths of the prior mean and variance, respectively. Then the posterior hyperparameter for $\sigma^2$ is

$$ \sigma^2_N = \frac{1}{\nu_0 + N} \left(\nu_0\sigma^2_0 + Ns^2 + \frac{N\kappa_0}{\kappa_0 + N}(\mu_0 - \bar x)^2\right)$$

where $s^2$ is the experimental variance. We can loosely rewrite this as

$$ \begin{align} \sigma^2_N = \frac{1}{\nu_0 + N} \left(\nu_0 \times \text{contribution from the prior} \\ + N\times \text{contribution from the experiment} \\ + \frac{N\kappa_0}{\kappa_0 + N}\times \text{uncertainty in $\mu$}\right) \end{align}$$

Answer 2

I do not understand the issue: for the model as described, there is a joint posterior distribution \begin{align} p(\mu, \sigma^2 | x,\mu_0, \sigma_0,\alpha, \beta) \propto \overbrace{p(\mu | \mu_0, \sigma_0)}^\text{Normal} \underbrace{p(\sigma^2 | \alpha, \beta)}_\text{inverse Gamma} \overbrace{p (x | \mu, \sigma^2)}^\text{Normal} \end{align} that is non-standard in the sense that the marginal posterior distributions of both $\mu$ and $\sigma^2$ are not traditional distributions and are not from exponential families. But this distribution on $(\mu, \sigma^2)$ can be computed at any value of $(\mu, \sigma^2,x,\mu_0, \sigma_0,\alpha, \beta)$ and hence can be simulated by all sorts of Monte Carlo approaches, including the or rather a Gibbs sampler.

Therefore, the lack of conjugacy does not prevent one from using a Gibbs sampler or another Markov Chain Monte Carlo method. They both produce simulations from the joint posterior $$p(\mu, \sigma^2 | x,\mu_0, \sigma_0,\alpha, \beta)$$ and not solely from the two marginal posteriors $$p(\mu | x,\mu_0, \sigma_0,\alpha, \beta)\quad\text{and}\quad p(\sigma^2 | x,\mu_0, \sigma_0,\alpha, \beta)$$

Answer 3

Stéphane is right that there's a semi-conjugacy, but you wouldn't lose conjugacy at all if you just expressed the prior and likelihood as distribution having three parameters that work out to be the mean, variance, and number of samples.