As you know, the likelihood of a multivariate Gaussian mixture model with C components and $d$ dimensional $x$ is as follow: $$p(x|\lambda) = \Sigma^{C}_{c=1}w_c p(x_t|\mu_c,\Sigma_c)$$ where $\lambda=\{w,\mu, \Sigma\}$, $w_c$ is the weight for component $c$ and $p(x_t|\mu_c,\Sigma_c)$ is Gaussian probability density function with mean $\mu_c$ and covariance matrix $\Sigma_c$. consequently, the log-likelihood would be: $$log(p(x|\lambda))=log(\Sigma^{C}_{c=1}w_c p(x_t|\mu_c,\Sigma_c))$$

now my question: is there any way to get rid of the log before the sum? It causes some numerical instabilities because of the exponential in the normal PDF, when the covariance matrix becomes very small. If log taken inside the sum, then:$$log(exp(X)) = X$$

I am looking for a solution for the general case for a $d$ dimensional $x$ and $C$ components GMM, with the PDF as follow:

PDF of a multivar. normal dist.

Any ideas?

  • 1
    $\begingroup$ Why is it causing instabilities? The probabilities are always strictly positive, because the pdf of the normal, as you mentioned, is an exponential, so even if some of the weights are zero, the sum will be strictly positive, and so the logarithm will be well-defined. Also, the logarithm is increasing and preserves concavity, so it is well behaved respect to the maximixation. $\endgroup$ – Anna SdTC Mar 14 '17 at 9:01
  • 1
    $\begingroup$ A possible solution is to look for the largest term in the sum for each $x_t$ say $w_1 p(x_t|\mu_1,\Sigma_1)$, and to get this term factor the sum,$$w_1 p(x_t|\mu_1,\Sigma_1)\sum_{c=1}^Cw_c p(x_t|\mu_c,\Sigma_c)\big/w_1 p(x_t|\mu_1,\Sigma_1)$$If this is not enough, you may have to find the maximum of the maxima over all $x_t$ and factor again by this term... $\endgroup$ – Xi'an Mar 14 '17 at 9:22
  • 1
    $\begingroup$ @Xi'an Usually in here, the trick is to come up with an extended version of the log-likelihood. There are cases that this is already available for the 2 dimensional case, where the formula is extended and the log is inside the sum over the components and consequently, log(exp()) will solve the overflow issue. But I couldn't find the general form where we could use a $d$ dimensional $x$. $\endgroup$ – PickleRick Mar 14 '17 at 9:49
  • 1
    $\begingroup$ @Xi'an thank you for your time, and your suggestion! your answer seems logical. I am going to put it to work and see how it performs... $\endgroup$ – PickleRick Mar 14 '17 at 10:07
  • 1
    $\begingroup$ @Anoosh: can you tell which part in the density is responsible for the underflow: is it the determinant $|\Sigma_c|$ or the quadratic form $(x_t-\mu_c)'\Sigma^{-1}_c(x-\mu_c)? $\endgroup$ – Xi'an Mar 14 '17 at 10:43

Switching logs and sums is not a possibility. When considering$$\sum_{t=1}^t \log\left[ \sum_{c=1}^C w_c p(x_t|\mu_c,\Sigma_c) \right]$$from an overflow-underflow perspective, you could

  1. ascertain whether or not exploring such extreme values for your parameters makes sense for the problem at hand. If not, treat extreme values as producing zero likelihoods;
  2. check if any of the determinants $|\Sigma_c|$ creates an indeterminacy issue and if so treat the determinant aside from the rest of the density by computing directly the log;
  3. determine which term $w_c p(x_t|\mu_c,\Sigma_c)$ is largest among all pairs $(t,c)$. For this you can look directly at the logarithms since$$\arg\max_{t,c} w_c p(x_t|\mu_c,\Sigma_c) =\arg\max_{t,c} \log\{w_c p(x_t|\mu_c,\Sigma_c)\}$$If this maximum creates an indeterminacy, then treat the whole likelihood as zero;
  4. remove this maximal value from all terms and treat underflow terms as zeros. If a given observation $x_t$ is such that all (log-)terms $\log\{w_c p(x_t|\mu_c,\Sigma_c)\}$ are indeterminate, then treat the whole likelihood as zero;
  5. check this impressive answer on Cross validated about the approximation of a Gaussian tail density by W. Huber if the problem persists.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.