Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Should the likelihood be increasing in every step of the EM algorithm?

I wrote an EM algorithm recently and the number it arrived does not seem to be the maximum.

I know this because I used the optim function on R on the same problem and it arrived parameters which gave a bigger log-likelihood value.

But even when I started at this maximum, I run my EM algorithm, it seem to converge to the other value I initially derived from my EM algorithm.

What is the most likely explanation for this?

share|improve this question
Please rephrase your first sentence correctly: The likelihood function is increasing at every step of the EM algorithm, as proved by Dempster, Laird and Rubin (1977). – Xi'an Dec 3 '12 at 21:15
Can you provide a minimally reproducible example? IE, some data, & the algorithm? (You may have a coding error.) – gung Dec 12 '12 at 15:37
up vote 1 down vote accepted

The estimator may increase or decrease during each iteration however the likelihood must increase.

You should make sure your likelihood is increasing at each step and see if you are converging to the same value.

share|improve this answer
Hi Glen, this is the question I posted the other day. I think I am almost done. Let the solution given by optim be called S1. Let the solution given by EM be called S2 S1 achieved better likelihood, but I fed S1 into the EM code, it converged to S2 again, this suggests there is something wrong with my algorithm? i.e. the log-likelihood was not increasing at every step? – Lost1 Dec 3 '12 at 14:24
There is an issue of converging to local maxima. I'm not sure what you mean by I fed S1 and got S2. Probably I problem with your code. Print the likelihood at each step and make sure it continually increases. – Glen Dec 3 '12 at 17:31

Perhaps not relevant in this case but note that if the E-step is estimated, with Monte Carlo methods or another approximation, it is possible for the likelihood to decrease.

A thought that might be relevant is that the EM does not converge towards the global maxima but to a local.

For more details see section 3 of "On the Convergence Properties of the EM Algorithm" by Wu.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.