Compute likelihood of mixture distribution while avoiding floating point problems

Question

I have a mixture distribution with likelihood function

$$ L(\theta) = \prod_{i=1}^N \sum_{k=1}^K f(X_i|\theta_k) \lambda_k $$

where $N$ is the sample size, $K$ is the number of component, $\theta_k$ is the parameter vector of component $k$, and $\lambda_k$ is the mixing probability of component $k$.

If I take the log of this I get

$$ \log \left[ L(\theta) \right] = \sum_{i=1}^N \log \left[ \sum_{k=1}^K f(X_i|\theta_k) \lambda_k \right] $$

My question is how do I evaluate the log-term for given $X$, $\theta$ and $\lambda$? Theoretically, I could compute the likelihoods $f(X_i|\theta_k)$ and sum them up weighted by the $\lambda_k$ and after take the log. However, I need to work with the log likelihoods to avoid floating point problems. My question therefore is how to compute the log term in the log likelihood above, if I can only compute $\log \left[ f(X_i|\theta_k) \right]$.

Answer 1

Consider all terms in $$\sum_{k=1}^K f(X_i|\theta_k) \lambda_k$$ namely $$(\log\{f(X_i|\theta_1) \lambda_1\},\ldots,\log\{f(X_i|\theta_K) \lambda_K\})$$ then identify the largest $$\log\{f(X_i|\theta_\zeta) \lambda_\zeta\}=\max(\log\{f(X_i|\theta_1) \lambda_1\},\ldots,\log\{f(X_i|\theta_K) \lambda_K\})$$ and write $$ \begin{aligned} \log\left\{\sum_{k=1}^K f(X_i|\theta_k) \lambda_k\right\} &=\log\{f(X_i|\theta_\zeta) \lambda_\zeta\} \\ &+\log\left\{ \sum_{k=1}^K \exp[\log\{f(X_i|\theta_k) \lambda_k\}-\log\{f(X_i|\theta_\zeta) \lambda_\zeta\}]\right\} \end{aligned} $$ Since all exponentiated terms are negative there should be no overflow issue.