1
$\begingroup$

I am reading through a document on learning Gaussian mixture models in Infer.NET. They assume the data is generated from 2 Gaussians where the prior distribution on means is Gaussian and the prior distribution on precisions is a Whishart distribution. The prior distribution on the mixture is a Dirichlet distribution. All of these priors are symmetric in the two Gaussians.

They do some inference on some data, and they get back that the posterior distribution on each of the two means is the same Gaussian. They then go on to talk about how to break the symmetry in the model so that the means can converge to different Gaussians.

How can it possibly be that the posteriors on the means are Gaussian? If I observe a million samples from a Gaussian Mixture Model (say unbeknownst to me the data is created by choosing with equal probability a normal distribution of mean 0 and variance 1 or a normal distribution with mean 100 and variance 1) it should be ABSOLUTELY CLEAR what the two means and standard deviations are. The symmetry of course means that the model doesn't know whether the first or the second Gaussian has mean 0 or mean 100, so shouldn't the posterior have two peaks, one near 0 and one near 100? If so, it's obviously not Gaussian.

I would appreciate any help in this matter.

$\endgroup$
  • 1
    $\begingroup$ TLDR; Could this be possibly related: stats.stackexchange.com/questions/178321/… ? $\endgroup$ – Tim Nov 3 '15 at 14:56
  • $\begingroup$ @time : Yes. It is exactly the label-switching problem I am talking about. Reading through the comments and papers in the link you gave makes it clear that the posteriors in this situation can have multiple modes and so, in general are not Gaussian. In my opinion, this exposes a bug in Infer.NET. $\endgroup$ – Josh Brown Kramer Nov 3 '15 at 17:18
  • $\begingroup$ TLDR: there is probably a misreading in here. It is well known that the posteriors on individual mixture components can have conjugate priors (including gaussians). $\endgroup$ – conjectures Jun 27 '17 at 11:31
1
$\begingroup$

The paper Bayesian Inference for Mixture: The Label Switching Problem says

A K-Component mixture distribution is invariant to permutations of the labels of the components. As a consequence, in a Bayesian framework, the posterior distribution of the mixture parameters has theoretically K! nodes.

To me, this answers the question : No. In general the posterior distribution is not Gaussian.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.