As an exercise I have implemented the EM algorithm for Gaussian mixtures, however, I have the problem that in high dimensions the densities of data points become so small that I get a numerical underflow.

Specifically, I am referring to calculating the "membership probabilities" in the E-step (called $T_{i,j}^{(t)}$ in this wikipedia article on the EM-algorithm). To obtain the membership probabilities of a data point I need to divide the density from each component by the sum of the densities from all components. For two mixture components and some fixed data point, I therefore need

$$p_i = \frac{\pi_i d_i}{\pi_1 d_1 + \pi_2 d_2},$$

where $i \in \{1,2\}$, $p_i$ is the membership probability of the data point at hand, $d_i$ is the density of the data point at hand, given the distribution of component $i$ with current estimates, and $\pi_i$ is the current estimate of the mixture proportion of component $i$.

Usually, when dealing with numerical underflow I take the log densities. However, then I don't see how to get back to the ratio $p_i$, because I can't get $\log(a + b)$ from $\log(a)$ and $\log(b)$.

I'm sure there must be a solution to this, since mixture models are commonly used in high dimensional settings. Can anybody help?

  • $\begingroup$ $$p_1={1}\Big/{1+\exp\{\log \pi_2 +\log d_1-\log \pi_1-\log d_1\}}$$ $\endgroup$ – Xi'an Apr 2 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.