I am reading through a document on learning Gaussian mixture models in Infer.NET. They assume the data is generated from 2 Gaussians where the prior distribution on means is Gaussian and the prior distribution on precisions is a Whishart distribution. The prior distribution on the mixture is a Dirichlet distribution. All of these priors are symmetric in the two Gaussians.
They do some inference on some data, and they get back that the posterior distribution on each of the two means is the same Gaussian. They then go on to talk about how to break the symmetry in the model so that the means can converge to different Gaussians.
How can it possibly be that the posteriors on the means are Gaussian? If I observe a million samples from a Gaussian Mixture Model (say unbeknownst to me the data is created by choosing with equal probability a normal distribution of mean 0 and variance 1 or a normal distribution with mean 100 and variance 1) it should be ABSOLUTELY CLEAR what the two means and standard deviations are. The symmetry of course means that the model doesn't know whether the first or the second Gaussian has mean 0 or mean 100, so shouldn't the posterior have two peaks, one near 0 and one near 100? If so, it's obviously not Gaussian.
I would appreciate any help in this matter.