I want to estimate the log likelihood of the joint distribution of 2 random variables, each follows a mixture of 2 Gaussian distribution. Is there any solution for it?


1 Answer 1


I believe that expectation maximization should address this problem. It is very similar to the analogous measurement error in two variables problems using binary latent variables instead of continuous latent variables.

If my stats theory is roughly right, the "marginal" mixture distribution likelihood can be stated as the integral of two "conditional" likelihoods over some unobserved, latent binary variable indicating membership to one particular likelihood. In the bivariate case, you have two latent binary variables, $b_1$ indicating whether the marginal $X$ belongs to a Gaussian likelihood, call it $X_1$, and $X_2$ with respective means $\mu_{1}$, $\mu_{2}$ and $b_2$ indicating whether the marginal $Y$ belongs to a Gaussian likelihood, call it $Y_1$, and $Y_2$ with respective means $\eta_{1}$, $\eta_{2}$.

In each case, knowing the latent variables, you can estimate the joint likelihood of bivariate Gaussian variables with linear regression.

$$L(X,Y | \mu_1, \mu_2, \Sigma) = \frac{1}{2\pi} \left| \Sigma \right|^{-1/2} \exp \left( -\frac{1}{2} (x-\mathbf{\mu})^T\Sigma^{-1}(x-\mathbf{\mu})\right)$$

At this point there are some considerations.. you need to consider whether the covariance can differ based on whether $b_1$ and $b_2$ agree or disagree. You could allow for 4, 3, 2, or 1 covariance(s) between each of the conditional $X$ and $Y$. I'll consider the most general case of having 4.

There are then 4 possible parameterizations of $L$ based on the values of $b_1$, and $b_2$ if the means are known. The bivariate Gaussian mean vectors $(\mu_1, \eta_1)$, $(\mu_2, \eta_1)$, $(\mu_1, \eta_2)$, $(\mu_2, \eta_2)$. Similarly there are 4 possible covariance matricies, call them $\Sigma_{1,1}$, $\Sigma_{2,1}$, $\Sigma_{1,2}$, and $\Sigma_{2,2}$ (not to be confused with matrix subsets of course). Similarly, each observation possibly contributes a corresponding likelihood which is denoted similarly $L_{1,1}$, $L_{1,2}$, $L_{2,1}$, and $L_{2,2}$. This gives the maximization step aspect of the EM algorithm.

The full (marginal) likelihood then is $$\int_{\left\{ 0,1 \right\} \times \left\{ 0,1 \right\} } (L_{1,1}^{b_1b_2}) (L_{1,2}^{b_1(1-b_2)}) (L_{2,1}^{(1-b_1)b_2} )(L_{2,2}^{(1-b_1)(1-b_2)}) $$

Using this, you can iteratively estimate the various parameters, and assign corresponding likelihoods.

  • $\begingroup$ For example, variable Z follows a 2-component mixture model (with 2 density f(z1), f(z2)), with the mixing probability $\pi$. And the likelihood can be defined as $f(z|z1,z2)= (1-\pi)f(z1)+\pi f(z2)$. How can I elaborate this adjusted likelihood in the bivariate case. $\endgroup$
    – pthao
    Commented Aug 7, 2014 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.