1
$\begingroup$

I'd like to estimate the parameters of a mixture model using MLE. The density is:

$$ f(x,y) = \mathcal{N}(x, y; \boldsymbol{\mu}, \Sigma) \cdot \alpha + \mathcal{N}(x; \mu, \sigma^2) \cdot \mathcal{U}_{[a, b]}(y) \cdot (1 - \alpha) $$

Where $\alpha$ is a mixing proportion to be learned, as will the parameters $\boldsymbol{\mu}, \Sigma, \mu, \sigma^2$.

I have previously used EM to fit a mixture of Gaussians, but I haven't previously tried to fit a mixture containing different distributions. Is there anything inherently wrong with what I'm trying to do?

Improve this question
$\endgroup$
6
  • $\begingroup$ Mixture distributions that aren't just all normal distributions are very common, and are no different in principle (if more cumbersome to track the parameters). But what might be an issue is the bounded interval of the uniform distribution. Your notation for the distributions is nonstandard, and it's unclear what the dots mean or what $b$ signifies for the uniform distribution -- could you put these in $\mathcal{N}(\,u,\sigma^2)$ and $\mathcal{U}_{[a,b]}$ form? $\endgroup$
    – jwimberley
    Nov 29, 2022 at 20:15
  • $\begingroup$ Sorry, the dots are meant to represent distribution parameters and simplify the density equation. I just wanted to convey which random variables were associated with which distributions. Above, $\mathcal{U}(b; \bullet)$ just meant evaluating the uniform density at $b$, and not referring to the bounds. $\endgroup$
    – student
    Nov 29, 2022 at 20:35
  • $\begingroup$ One problem is that the dots suggest relationships among the parameters. Moreover, the use of "$\alpha$" along with the dots makes the role of $\alpha$ ambiguous: is it being estimated or stipulated? These ambiguities suggest you might get conflicting answers that vary according to how they are interpreted--and that will make everyone unhappy. $\endgroup$
    – whuber ♦
    Nov 29, 2022 at 20:43
  • $\begingroup$ I didn't consider that, thank you. Hopefully this most recent edit is an improvement. $\endgroup$
    – student
    Nov 29, 2022 at 20:51
  • 1
    $\begingroup$ Ah sorry, the first Normal is bivariate. I'll fix that. Yes the interval $[a,b]$ is given and comes from domain knowledge. Without revealing too much, there is a fixed region where an event is more likely. $\endgroup$
    – student
    Nov 29, 2022 at 21:15

1 Answer 1

Reset to default
0
$\begingroup$

There's nothing special about mixtures of normal distributions that enable the EM algorithm to work, in principle, and a mixture of other sorts of distributions is equally amenable. In practice, other sorts of mixtures may be more cumbersome and computationally harder to work with -- especially in regards to the maximization step of the EM algorithm.

In the gaussian mixture case, the M-step has a closed form solution; this is true in most cases where the mixture is one of exponential-family distributions (see references here). In the general case, though, the maximization step may require numerical optimization, which could have large practical consequences for the complexity of the algorithm.

The worst case would be when the densities are not differentiable functions of all the parameters being optimized. For one, first (let alone second) order optimization methods could not be applied to the maximization step. But more importantly, I am uncertain under what conditions the EM algorithm is guaranteed to converge when the densities are not smooth (or at least continuous) functions of all the variable parameters.

In the example given in the question, luckily, the density is indeed a differentiable function of all the parameters, since the lower and upper bounds $[a,b]$ of the uniform distribution are fixed a priori, and so my uncertainties I voiced in the above paragraph do not apply. Since the mixture is not made up entirely of exponential family distributions, it's likely to fall into the intermediate case in which the M-step requires numerical optimization and is more costly.

One thing I do note is that distribution in question can be marginalized across either the $x$ or $y$ axes:

$$ f(x) = \int f(x,y) \, dy = \mathcal{N}(x;\mu_x,\sigma_x^2) \cdot \alpha + \mathcal{N}(x;\mu,\sigma^2) \cdot (1-\alpha) \\ f(y) = \int f(x,y) \, dy = \mathcal{N}(y;\mu_y,\sigma_y^2) \cdot \alpha + \mathcal{U}_{[a,b]}(y) \cdot (1-\alpha) $$ This seems like a feature that one could take advantage of -- if only to use the marginal distribution along $y$ and/or $x$ to get initial estimates of $\alpha$, $\mu$, $\sigma^2$, $\mu_x$, $\mu_y$, $\sigma_x^2$, and $\sigma_y^2$, and then use these as the starting point in the full 2D optimization. The marginal $x$ distribution is just a standard gaussian mixture to which the efficient out-of-the-box methods can be applied; the marginal $y$ distribution also looks quite tacklable. The full 2D optimization introduces only one additional parameter is the correlation coefficient $\rho$ in the bivariate normal (the matrix $\Sigma$ being parametrized by $\sigma_x^2$, $\sigma_y^2$, and $\rho$).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Just one note -- naturally any point with $y_i$ outside the range $[a,b]$ cannot, by definition, have arisen from the component with the uniform distribution in that range. Allowing it a latent variable with a probability $0$ of having arisen from this component would lead to logarithms of zero in the maximization step -- better instead to just treat those as having a known, rather than non-latent, component. $\endgroup$
    – jwimberley
    Nov 30, 2022 at 22:56
  • $\begingroup$ Actually it works out more simply than I expected -- the E step will never assign non-zero probabilities that points outside $[a,b]$ lie in the second component. So formally speaking the log term in the $Q$ function has some terms like $0 \log 0$, which can just be ignored, and the M step gives closed form updates that are no different from the normal gaussian mixture case, save that there's one less gaussian than normal. $\endgroup$
    – jwimberley
    Dec 1, 2022 at 2:42
  • $\begingroup$ Thank you for the answer and additional insights on how to implement EM for this mixture! $\endgroup$
    – student
    Dec 1, 2022 at 14:22
  • $\begingroup$ If you have a moment, could you please take a look at another mixture question that I have? $\endgroup$
    – student
    Feb 8 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.