Let's assume that we generate some values by a mixture of two Gaussians. Now we want to find the parameters of the two Gaussians by likelihood maximisation. One good expect that the optimisation will find the two original Gaussians if amount of generated data is sufficient.
However, I observe the following behaviour. One of the Gaussians fits the two components of the distributions while the other one finds a single point and becomes narrower and narrower around this point.
Since the second Gaussian is very narrow (and becomes more and more narrower), the corresponding point get a huge probability density and the total likelihood goes to infinity.
So, obviously the procedure fails here. However, it is very surprising, since:
- I have a huge amount of the synthetic data.
- The model is very simple.
- The fitted model reflects the real generator perfectly (the data is generated by a mixture of two Gaussians).
So, my question is: Is the described problem well known and is there a standard solution to it?