Let's say that I have a mixture of Gaussians representing a likelihood:

$$ p(\mathbf{x}) = \sum_{i=1}^K\phi_i \mathcal{N}(\boldsymbol{\mu_i,\Sigma_i}) $$

What is the posterior distribution given a prior Gaussian distribution $\mathcal{N}(\boldsymbol{\mu_p,\Sigma_p})$?

Is it simply this? $$ p(\mathbf{x}) = \sum_{i=1}^K\phi_i \mathcal{N}(\boldsymbol{\mu_i,\Sigma_i})\mathcal{N}(\boldsymbol{\mu_p,\Sigma_p})\\ = \sum_{i=1}^K\phi_i \mathcal{N}(\boldsymbol{\mu_{i,p},\Sigma_{i,p}}) $$ with:

$$ \Sigma_{i,p} = \Sigma_i(\Sigma_i + \Sigma_p)^{-1}\Sigma_p\\ \mu_{i,p} = \Sigma_p(\Sigma_i + \Sigma_p)^{-1}\mu_i + \Sigma_i(\Sigma_i + \Sigma_p)^{-1}\mu_p $$

  • 2
    $\begingroup$ The "new" Gaussian is a prior for what? $\endgroup$ – Tim May 6 '15 at 10:15
  • $\begingroup$ the posterior is the result of the multiplication of the probability distributions. The prior is an assumption that I make, and the initial mixture of gaussians is inferred from data. $\endgroup$ – fstab May 6 '15 at 10:23
  • 2
    $\begingroup$ Yes, I know, but this prior is your assumption about what? $\endgroup$ – Tim May 6 '15 at 10:30
  • $\begingroup$ locations, expressed in 2d coordinates (hence the gaussians here are bi-variate) $\endgroup$ – fstab May 6 '15 at 10:34
  • $\begingroup$ So you are asking about conjugate Normal prior for Normal distribution with known $\sigma^2$, see en.wikipedia.org/wiki/Conjugate_prior or cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf or stat.duke.edu/courses/Fall10/sta290/Lectures/Normal/… for examples. $\endgroup$ – Tim May 6 '15 at 10:38

If I understand your question correctly, you are asking: "What is the conjugate prior for a Mixture of Gaussians likelihood?".

If that is the case, and assuming your covariance (or precision) matrix is fixed, you would use:

  • Dirichlet distribution for mixture weights (for a two mixture model, this reduces to Beta)

  • Gaussian densities for means

So, I guess you are missing the mixture distribution (multinomial) in your equations. These slides cover the topic, for cases when your covariance (or precision) matrix is not fixed (As @Tim points out in the comments). Also, this paper is relevant.

| cite | improve this answer | |
  • $\begingroup$ You cannot use Gaussian as a prior for variance since Gaussian distribution ranges from $-\infty$ to $\infty$ while variance can be only >0 ! $\endgroup$ – Tim May 6 '15 at 10:32
  • $\begingroup$ @Tim Correct, you would use a gamma distribution as a conjugate prior for precision (only). But that is not what I meant in my answer, will edit. $\endgroup$ – Zhubarb May 6 '15 at 10:35
  • $\begingroup$ Not that this matters, but for the multivariate normal, the prior is Wishart (the multivariate generalization) of the gamma distribution. $\endgroup$ – Neil G Oct 25 '16 at 12:42

Yes, your equations are correct — except you need to update your mixture probabilities to account for your prior contradicting some of them.

For each mode, calculate the integral of pointwise product of densities of prior and the mode. Multiply that by that mixture's probability. Then renormalize all of the probabilities.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.