Termination Condition(s) for Expectation Maximization

Question

What are good criteria for deciding when to terminate the expectation-maximization algorithm.

I know that the idea is that you should terminate when the change in the data log likelihood is "small" or the change in the model parameters is "small" over one (or a few) iterations. But how do you determine "small"?

So far, my literature search has only provided papers that prove convergence under different conditions, or provide specialized techniques to improve the rate of convergence. I've not seen good, practical, results on when to terminate the algorithm in practical use.

Answer 1

Practically speaking, you set the change of likelihood, for instance, the 1e-3 defaulted in sklearn, or/and you set the max iteration as 100. And you try several different initializations, to try to prevent converging to local optimum points, and choose the best model.

If time and computation permit you can enlarge the iteration or/and make the likelihood change threshold smaller, like 1e-4 or the like.

Answer 2

You can use an absolute error criterion (e.g. difference between successive iterations $|\ell(\theta_{n+1}) - \ell(\theta_n)|< 10^{-4}$). You can also choose a relative error criterion, e.g. $\frac{|\ell(\theta_{n+1}) - \ell(\theta_n)|}{|\ell(\theta_n)|}< 10^{-4}$. This is somewhat more reasonable because you do not really know how large or small your log-likelihood is and normalizing makes your stopping condition independent of scale. Just make sure you do not set the desired tolerance too small or you will never achieve it.