3
$\begingroup$

Let's say that I have a mixture of Gaussians representing a likelihood:

$$ p(\mathbf{x}) = \sum_{i=1}^K\phi_i \mathcal{N}(\boldsymbol{\mu_i,\Sigma_i}) $$

What is the posterior distribution given a prior Gaussian distribution $\mathcal{N}(\boldsymbol{\mu_p,\Sigma_p})$?

Is it simply this? $$ p(\mathbf{x}) = \sum_{i=1}^K\phi_i \mathcal{N}(\boldsymbol{\mu_i,\Sigma_i})\mathcal{N}(\boldsymbol{\mu_p,\Sigma_p})\\ = \sum_{i=1}^K\phi_i \mathcal{N}(\boldsymbol{\mu_{i,p},\Sigma_{i,p}}) $$ with:

$$ \Sigma_{i,p} = \Sigma_i(\Sigma_i + \Sigma_p)^{-1}\Sigma_p\\ \mu_{i,p} = \Sigma_p(\Sigma_i + \Sigma_p)^{-1}\mu_i + \Sigma_i(\Sigma_i + \Sigma_p)^{-1}\mu_p $$

$\endgroup$
6
  • 2
    $\begingroup$ The "new" Gaussian is a prior for what? $\endgroup$
    – Tim
    May 6, 2015 at 10:15
  • $\begingroup$ the posterior is the result of the multiplication of the probability distributions. The prior is an assumption that I make, and the initial mixture of gaussians is inferred from data. $\endgroup$
    – fstab
    May 6, 2015 at 10:23
  • 2
    $\begingroup$ Yes, I know, but this prior is your assumption about what? $\endgroup$
    – Tim
    May 6, 2015 at 10:30
  • $\begingroup$ locations, expressed in 2d coordinates (hence the gaussians here are bi-variate) $\endgroup$
    – fstab
    May 6, 2015 at 10:34
  • $\begingroup$ So you are asking about conjugate Normal prior for Normal distribution with known $\sigma^2$, see en.wikipedia.org/wiki/Conjugate_prior or cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf or stat.duke.edu/courses/Fall10/sta290/Lectures/Normal/… for examples. $\endgroup$
    – Tim
    May 6, 2015 at 10:38

2 Answers 2

1
$\begingroup$

If I understand your question correctly, you are asking: "What is the conjugate prior for a Mixture of Gaussians likelihood?".

If that is the case, and assuming your covariance (or precision) matrix is fixed, you would use:

  • Dirichlet distribution for mixture weights (for a two mixture model, this reduces to Beta)

  • Gaussian densities for means

So, I guess you are missing the mixture distribution (multinomial) in your equations. These slides cover the topic, for cases when your covariance (or precision) matrix is not fixed (As @Tim points out in the comments). Also, this paper is relevant.

$\endgroup$
3
  • $\begingroup$ You cannot use Gaussian as a prior for variance since Gaussian distribution ranges from $-\infty$ to $\infty$ while variance can be only >0 ! $\endgroup$
    – Tim
    May 6, 2015 at 10:32
  • $\begingroup$ @Tim Correct, you would use a gamma distribution as a conjugate prior for precision (only). But that is not what I meant in my answer, will edit. $\endgroup$
    – Zhubarb
    May 6, 2015 at 10:35
  • $\begingroup$ Not that this matters, but for the multivariate normal, the prior is Wishart (the multivariate generalization) of the gamma distribution. $\endgroup$
    – Neil G
    Oct 25, 2016 at 12:42
0
$\begingroup$

Yes, your equations are correct — except you need to update your mixture probabilities to account for your prior contradicting some of them.

For each mode, calculate the integral of pointwise product of densities of prior and the mode. Multiply that by that mixture's probability. Then renormalize all of the probabilities.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.