What is the correct procedure to fit finite mixture models on data with many duplicate values using EM? Let's say I have N(0,1) distributed data and try to fit a 2 component mixture using EM. There are more zeros than excepted if the data was truly from N(0,1). One of the components ends up being N(0,1), but the other basically contains all the zeros and has infinite likelihood since the values are all duplicates (all 0). My goal is basically to show a 1 component model fits better but likelihood is infinite for the 2 component model. What should I do in this scenario? I've tried smoothing the assignment probabilities so the MLE on the component containing zeros contains the other data as well with very low probability but this is extremely sensitive to the smoothing parameter. Are there better alternatives?
Firstly, within the scope you've set, I think the EM algorithm does the right thing. A normal distribution cannot have lots of values that are exactly the same value - the only way to make that occur is to set the mean of one mixture component to that value and the variance to zero (the second mixture component you are presumably getting).
If your zero values are truly exactly zero, then a point distribution at zero (basically a $N(0,\epsilon)$ as $\epsilon \to 0$) and a normal as a mixture may well be the best way to describe this. In that scenario, showing that a single component normal is "better" is futile, because it clearly is not (it just cannot have lots of values that are truly exactly zero).
If the zero values are not truly zeros, but perhaps only zeros up to rounding, then including the interval censored nature of the data (i.e. use the likelihood for interval censored normal data) might well lead to a fit that has two mixture components with non-zero variance.
If you want to force the variance of both components to be non-zero, there's also the option of going down a full Bayesian approach and to specify proper priors on the variances - but, again, that might be sensitive to the choice of the prior distributions.