Likelihood of the joint distribution of 2 Gaussian mixtures

Question

I want to estimate the log likelihood of the joint distribution of 2 random variables, each follows a mixture of 2 Gaussian distribution. Is there any solution for it?

Answer 1

I believe that expectation maximization should address this problem. It is very similar to the analogous measurement error in two variables problems using binary latent variables instead of continuous latent variables.

If my stats theory is roughly right, the "marginal" mixture distribution likelihood can be stated as the integral of two "conditional" likelihoods over some unobserved, latent binary variable indicating membership to one particular likelihood. In the bivariate case, you have two latent binary variables, $b_1$ indicating whether the marginal $X$ belongs to a Gaussian likelihood, call it $X_1$, and $X_2$ with respective means $\mu_{1}$, $\mu_{2}$ and $b_2$ indicating whether the marginal $Y$ belongs to a Gaussian likelihood, call it $Y_1$, and $Y_2$ with respective means $\eta_{1}$, $\eta_{2}$.

In each case, knowing the latent variables, you can estimate the joint likelihood of bivariate Gaussian variables with linear regression.

$$L(X,Y | \mu_1, \mu_2, \Sigma) = \frac{1}{2\pi} \left| \Sigma \right|^{-1/2} \exp \left( -\frac{1}{2} (x-\mathbf{\mu})^T\Sigma^{-1}(x-\mathbf{\mu})\right)$$

At this point there are some considerations.. you need to consider whether the covariance can differ based on whether $b_1$ and $b_2$ agree or disagree. You could allow for 4, 3, 2, or 1 covariance(s) between each of the conditional $X$ and $Y$. I'll consider the most general case of having 4.

There are then 4 possible parameterizations of $L$ based on the values of $b_1$, and $b_2$ if the means are known. The bivariate Gaussian mean vectors $(\mu_1, \eta_1)$, $(\mu_2, \eta_1)$, $(\mu_1, \eta_2)$, $(\mu_2, \eta_2)$. Similarly there are 4 possible covariance matricies, call them $\Sigma_{1,1}$, $\Sigma_{2,1}$, $\Sigma_{1,2}$, and $\Sigma_{2,2}$ (not to be confused with matrix subsets of course). Similarly, each observation possibly contributes a corresponding likelihood which is denoted similarly $L_{1,1}$, $L_{1,2}$, $L_{2,1}$, and $L_{2,2}$. This gives the maximization step aspect of the EM algorithm.

The full (marginal) likelihood then is $$\int_{\left\{ 0,1 \right\} \times \left\{ 0,1 \right\} } (L_{1,1}^{b_1b_2}) (L_{1,2}^{b_1(1-b_2)}) (L_{2,1}^{(1-b_1)b_2} )(L_{2,2}^{(1-b_1)(1-b_2)}) $$

Using this, you can iteratively estimate the various parameters, and assign corresponding likelihoods.