# Using the EM Algorithm for unimodal distributions?

I've really only seen EM used for mixtures where one can point out multiple modes visually - e.g, the classic mixture of gaussians example. I would like to use EM for a mixture of an empirically defined, sharply peaked distribution and something that is more uniform - does anyone have an idea as to how much confidence I should put in the resulting estimates, or prior experience with similar applications?

• Re your e.g.: It's a common misconception that a mixture of two Gaussians is necessarily bimodal. A mixture of two Gaussians with equal standard deviations is bimodal only if the difference in their means is at least twice their standard deviation. See: Schilling, Mark F.; Watkins, Ann E.; Watkins, William (2002). "Is Human Height Bimodal?". The American Statistician 56(3): 223–229. dx.doi.org/10.1198/00031300265 – onestop Nov 3 '10 at 7:15

There are two questions here: 1) how much confidence should you put in your model with peaked and flat components. 2) how much confidence should you put in the EM algorithm as a way to fit this model.

Question 1 has the same answers as any other model, e.g. a regression model with particular covariates or a factor analysis model with a certain number of factors. The only specific consideration I can think of is that you may be introducing a the alternative flat data generating source to make a more robust estimate of peaked data source. This is a standard noisy measurement model. For comparison, you might also work with a fatter tailed peaked source, e.g. T-distribution vs. Normal + Uniform.

As for Question 2, EM is just a maximum likelihood method. This means, first that there may be better parameter values available, because it may have found a local minimum, and second that you can get degenerate solutions because there is no prior / regularization in the setup. Both are standard ML problems, not really anything to do with the EM algorithm, although the latter is probably made slightly worse by having missing data in the mix.

For a more elaborate discussion, see McLachlan and Krishnan's 'EM Algorithm and Extensions'.

I have used various algorithms, including Bayesian approaches (and, I am sorry to confess, even Excel many years ago), to fit mixtures. When there is not a clear visual indication of the two (or more components) in the histogram, you can expect the likelihood function to be extremely flat--almost parabolic--near its peak. This is because the visual impression translates mathematically into an ability to trade off some proportion of one mixture with an equivalent proportion of the other (adjusting the parameters of the components to keep a good fit) while making only a minor change to the likelihood. In many cases it's difficult to pin down the maximum likelihood. (This is evidenced by regime-switching in the Markov chains, for instance: a chain will pursue an area where one component predominates and after longish periods switch to an area where another component predominates, never really settling down to a single optimum.) In any event you also want to assess uncertainty. This is reflected by how much change is needed in the mixture parameters to reduce the likelihood by some threshold amount. The near-parabolic flatness near the optimum delineates a long "ridge" of near-optimum values, resulting in a long elliptical confidence region for the mixture. Usually the major axis of that ellipse corresponds to the mixture proportions. Thus, you might conclude that your data are $p$ percent of component A and $1-p$ percent of component B, but $p$ might be anywhere from 0 to 70%. (Yes, there are boundary value problems with mixtures, too.) It can take an extraordinary amount of data to reduce these wide confidence intervals if you can even reliably find them.

These problems are exacerbated when only the tails of the data provide most of the information needed to disentangle the distributions. This would often be the case for unimodal data.

I have a paper in press that explores application of EM to estimation of a Von Mises & uniform mixture in the circular domain. (The Von Mises is the circular analogue of a gaussian.)

• That's nice, Mike, but what does the paper say? – whuber Nov 3 '10 at 14:17
• @whuber: possibly you missed the link to the paper posted in my response. If not, possibly you didn't bother to look at it, in which case the gist is that I find that the EM algorithm performs well in some cases (lots of data, reasonable proportion of reasonably concentrated Von Mises data), not so well in others (little data, low proportion of Von Mises, Von Mises with low concentration). – Mike Lawrence Nov 4 '10 at 12:30
• This issue has come up on other SE sites, too. The point is that answers should be stand-alone, for many reasons. The intent is to make our threads a source of information, not just some gateway of links to possibly evanescent material. The response "an answer is at such-and-such a place" is appropriate when the answer is simple, standard, and easily reachable from a search (e.g., has a good Wikipedia article). Otherwise, especially when the answer reflects a significant investigation like yours, please at least provide a summary. There's no need to attack me for making this reminder. – whuber Nov 4 '10 at 14:05