0
$\begingroup$

To maximize the likelihood of a mixture model with unobserved latent variables, the Expectation Maximization is conventionally applied. Assuming we have data $x_1,\dots,x_n$ from a fixed number of distributions, with corresponding density functions $f_j(x;\theta_j)$. Now let the variables $z_{1j},\dots,z_{nj}$ be the binary variable denoting if the corresponding observation is a random outcome of distribution $j$. Now, the full likelihood of observation $i$ is often formulated as follows

$L(\theta;x_i,z)=\prod_{j}f_j(x_i;\theta_j)^{z_{ij}}$ with log-likelihood

$l(\theta;x_i,z)=\sum_{j}{z_{ij}}\ln(f_j(x_i;\theta_j))$

In the E-step of the EM algorithm, the latent variable $z_{ij}$ is estimated w.r.t current parameters. Now, what happens if observation $x_i$ belongs to distribution $j$ and is such that for some other distribution $k\neq j$, $f_k(x_i,\theta_k)=0$. Then the log-likelihood will take value $-\infty$.

I understand $E({z_{ik}}|x_i,\theta_k)$ must take value 0 in this case, and thus "removing" this term from the summation. However, in most programming languages $0\cdot \infty$ is ill defined (I believe, for instance R returns NaN), so won't this create convergence issues? Is there a standard procedure to deal with this?

I appreciate all comments/references to literature and answers.

$\endgroup$
2
$\begingroup$

For those reading at home who are worried by the whole $0 \times \infty = 0$, it's reasonable to accept that if $z_{ij} = 0$ then $z_{ij} \log (f_j(x_i, \theta_j)) = 0$ beyond the mere conveniece of it. To see this, note that

$ z_{ij} = \frac{f_j(x_i, \theta_j)}{\sum_k f_k(x_i, \theta_k)} $

and that if we take the limiting value of

$ \frac{f_j(x_i, \theta_j)}{\sum_k f_k(x_i, \theta_k)} \log (f_j(x_i, \theta_j) ) $

as $f_j(x_i, \theta_j) \rightarrow 0^+$, we get the answer zero* provided there exists at least one other $f_k(x_i, \theta_k) > 0$.

From a programming point of view I'm not aware of any convention on how to deal with this. Some languages will define $0^0 = 1$ even if $0 \log (0)$ is not defined (matlab, for example), so you could evaluate $\log (f_j(x_i, \theta_j)^{z_{ij}})$ instead of $z_{ij} \log (f_j(x_i, \theta_j))$ and avoid NaNs. This gives nice clean looking code, but is problematic for a number of other reasons (you replace a multiplication with an exponentiation, and I think it can generate precision issues).

It's best really to just have an If statement that checks whether each $z_{ij} == 0$, and skips evaluating that log likelihood entirely if so. Alternatively, you could evaluate $z_{ij} \log( \max (f_j(x_i, \theta_j), \epsilon) )$ where $\epsilon$ is the smallest value which your logarithm gives finite answers for. If $z_{ij}=0$, you'll get the same result.

*The easiest way to check this is to consider the limit of $ \frac{\log (a)}{ 1/a} $ as $a \rightarrow 0$. As both numerator and denominator tend to $\infty$ we can just apply L'Hopital's rule.

$\endgroup$
  • $\begingroup$ Thank you, I've already tested both solutions you suggested and as a result the estimator at least yields a real number. I was however hoping this would result in convergence to a better solution (since the function could not be evaluated at the theoretical parameters of my simulated data), which it did not. I believe this is a consequence of convergence to local maxima/minima instead... $\endgroup$ – Good Guy Mike Mar 9 '15 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.