Convergence of EM in Mixture Models w.r.t unlikely events $(f(\cdot)=0)$ in either distribution

To maximize the likelihood of a mixture model with unobserved latent variables, the Expectation Maximization is conventionally applied. Assuming we have data $x_1,\dots,x_n$ from a fixed number of distributions, with corresponding density functions $f_j(x;\theta_j)$. Now let the variables $z_{1j},\dots,z_{nj}$ be the binary variable denoting if the corresponding observation is a random outcome of distribution $j$. Now, the full likelihood of observation $i$ is often formulated as follows

$L(\theta;x_i,z)=\prod_{j}f_j(x_i;\theta_j)^{z_{ij}}$ with log-likelihood

$l(\theta;x_i,z)=\sum_{j}{z_{ij}}\ln(f_j(x_i;\theta_j))$

In the E-step of the EM algorithm, the latent variable $z_{ij}$ is estimated w.r.t current parameters. Now, what happens if observation $x_i$ belongs to distribution $j$ and is such that for some other distribution $k\neq j$, $f_k(x_i,\theta_k)=0$. Then the log-likelihood will take value $-\infty$.

I understand $E({z_{ik}}|x_i,\theta_k)$ must take value 0 in this case, and thus "removing" this term from the summation. However, in most programming languages $0\cdot \infty$ is ill defined (I believe, for instance R returns NaN), so won't this create convergence issues? Is there a standard procedure to deal with this?

I appreciate all comments/references to literature and answers.

For those reading at home who are worried by the whole $0 \times \infty = 0$, it's reasonable to accept that if $z_{ij} = 0$ then $z_{ij} \log (f_j(x_i, \theta_j)) = 0$ beyond the mere conveniece of it. To see this, note that

$z_{ij} = \frac{f_j(x_i, \theta_j)}{\sum_k f_k(x_i, \theta_k)}$

and that if we take the limiting value of

$\frac{f_j(x_i, \theta_j)}{\sum_k f_k(x_i, \theta_k)} \log (f_j(x_i, \theta_j) )$

as $f_j(x_i, \theta_j) \rightarrow 0^+$, we get the answer zero* provided there exists at least one other $f_k(x_i, \theta_k) > 0$.

From a programming point of view I'm not aware of any convention on how to deal with this. Some languages will define $0^0 = 1$ even if $0 \log (0)$ is not defined (matlab, for example), so you could evaluate $\log (f_j(x_i, \theta_j)^{z_{ij}})$ instead of $z_{ij} \log (f_j(x_i, \theta_j))$ and avoid NaNs. This gives nice clean looking code, but is problematic for a number of other reasons (you replace a multiplication with an exponentiation, and I think it can generate precision issues).

It's best really to just have an If statement that checks whether each $z_{ij} == 0$, and skips evaluating that log likelihood entirely if so. Alternatively, you could evaluate $z_{ij} \log( \max (f_j(x_i, \theta_j), \epsilon) )$ where $\epsilon$ is the smallest value which your logarithm gives finite answers for. If $z_{ij}=0$, you'll get the same result.

*The easiest way to check this is to consider the limit of $\frac{\log (a)}{ 1/a}$ as $a \rightarrow 0$. As both numerator and denominator tend to $\infty$ we can just apply L'Hopital's rule.

• Thank you, I've already tested both solutions you suggested and as a result the estimator at least yields a real number. I was however hoping this would result in convergence to a better solution (since the function could not be evaluated at the theoretical parameters of my simulated data), which it did not. I believe this is a consequence of convergence to local maxima/minima instead... – Good Guy Mike Mar 9 '15 at 11:19