Maximum likelihood estimate for mixture of different distributions I'd like to estimate the parameters of a mixture model using MLE. The density is:
$$
f(x,y) = \mathcal{N}(x, y; \boldsymbol{\mu}, \Sigma) \cdot \alpha + \mathcal{N}(x; \mu, \sigma^2) \cdot \mathcal{U}_{[a, b]}(y) \cdot (1 - \alpha)
$$
Where $\alpha$ is a mixing proportion to be learned, as will the parameters $\boldsymbol{\mu}, \Sigma, \mu, \sigma^2$.
I have previously used EM to fit a mixture of Gaussians, but I haven't previously tried to fit a mixture containing different distributions. Is there anything inherently wrong with what I'm trying to do?
 A: There's nothing special about mixtures of normal distributions that enable the EM algorithm to work, in principle, and a mixture of other sorts of distributions is equally amenable. In practice, other sorts of mixtures may be more cumbersome and computationally harder to work with -- especially in regards to the maximization step of the EM algorithm.
In the gaussian mixture case, the M-step has a closed form solution; this is true in most cases where the mixture is one of exponential-family distributions (see references here). In the general case, though, the maximization step may require numerical optimization, which could have large practical consequences for the complexity of the algorithm.
The worst case would be when the densities are not differentiable functions of all the parameters being optimized. For one, first (let alone second) order optimization methods could not be applied to the maximization step. But more importantly, I am uncertain under what conditions the EM algorithm is guaranteed to converge when the densities are not smooth (or at least continuous) functions of all the variable parameters.
In the example given in the question, luckily, the density is indeed a differentiable function of all the parameters, since the lower and upper bounds $[a,b]$ of the uniform distribution are fixed a priori, and so my uncertainties I voiced in the above paragraph do not apply. Since the mixture is not made up entirely of exponential family distributions, it's likely to fall into the intermediate case in which the M-step requires numerical optimization and is more costly.
One thing I do note is that distribution in question can be marginalized across either the $x$ or $y$ axes:
$$
f(x) = \int f(x,y) \, dy = \mathcal{N}(x;\mu_x,\sigma_x^2) \cdot \alpha + \mathcal{N}(x;\mu,\sigma^2) \cdot (1-\alpha) \\
f(y) = \int f(x,y) \, dy = \mathcal{N}(y;\mu_y,\sigma_y^2) \cdot \alpha + \mathcal{U}_{[a,b]}(y) \cdot (1-\alpha)
$$
This seems like a feature that one could take advantage of -- if only to use the marginal distribution along $y$ and/or $x$ to get initial estimates of $\alpha$, $\mu$, $\sigma^2$, $\mu_x$, $\mu_y$, $\sigma_x^2$, and $\sigma_y^2$, and then use these as the starting point in the full 2D optimization. The marginal $x$ distribution is just a standard gaussian mixture to which the efficient out-of-the-box methods can be applied; the marginal $y$ distribution also looks quite tacklable. The full 2D optimization introduces only one additional parameter is the correlation coefficient $\rho$ in the bivariate normal (the matrix $\Sigma$ being parametrized by $\sigma_x^2$, $\sigma_y^2$, and $\rho$).
