# References that justify use of Gaussian Mixtures

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much complexity. There are a few analytic properties we should expect to hold which are not clear in general. In particular:

• Say $S_n$ is the class of all Gaussian mixtures with $n$ components. For any continuous distribution $P$ on the reals, are we guaranteed that as $n$ grows, we can approximate $P$ with a GMM with negligible loss in the sense of relative entropy? That is, does $$\lim_{n\rightarrow \infty}\inf_{\hat{P}\in S_n} D(P||\hat{P})=0?$$
• Say we have a continuous distribution $P$ and we have found an $N$-component Gaussian mixture $\hat{P}$ which is close to $P$ in total variation: $\delta(P,\hat{P})<\varepsilon$. Can we bound $D(P||\hat{P})$ in terms of $\epsilon$?
• If we want to observe $X\sim P_X$ through independent additive noise $Y\sim P_Y$ (both real, continuous), and we have GMMs $\hat{X} \sim Q_X, \hat{Y} \sim Q_N$ where $\delta(P,Q)<\epsilon$, then is this value small: $$\left|\mathsf{mmse}(X|X+Y)-\mathsf{mmse}(\hat{X}| \hat{X}+\hat{Y})\right|,$$ i.e. is it true that estimating $X$ through $Y$ noise is about as hard as estimating $\hat{X}$ through $\hat{Y}$ noise?
• Can you do it for non-additive noise models like Poisson noise?

My (short) literature review so far has just turned up very applied tutorials. Does anyone have any references that rigorously demonstrate under what conditions we are justified in using mixture models?

• The set of GMMs is dense in the set of distributions in the weak topology (corresponding to convergence in distribution); see e.g. here. I'm not sure whether your first statement holds, though it would certainly require allowing zero-variance components in the mixture to deal with any point masses in $P$. I'm also skeptical about the second bullet point, again because of the issue of point masses. – Dougal Jan 27 '16 at 3:23
• Good point, I've specified everything should be continuous – enthdegree Jan 27 '16 at 3:34
• You might have better luck looking at the literature on kernel density estimation with Gaussian kernels. Since you have a mixture of Gaussians with one per sample, as the number of samples goes up, do you get an asymptotically unbiased and consistent estimator of the distribution? I think the answer is yes, but couldn't immediately find a reference. – Greg Ver Steeg Jan 28 '16 at 18:37
• @enthdegree: Very good question. Because you want to use strong topologies (KL divergence and total-variation), the general answer to your first two points is no: for example, consider a fat-tailed distribution; The KL to any finite gaussian mixture is infinite (I'm pretty sure this works, though not 100%). But this leads to the much more interesting question, for which subclass of probability distributions would all your bullet points apply ? I don't know the answer but it seems extremely interesting. My guess is its probably almost all probability distributions. – Guillaume Dehaene Feb 3 '16 at 13:30
• I took a class with this book. link It does some decent background on fundamentals. – EngrStudent - Reinstate Monica Feb 4 '16 at 21:10

In econometrics, where the context is of mixture distributions of coefficients in logit models, the standard reference is: MIXED MNL MODELS FOR DISCRETE RESPONSE DANIEL MCFADDEN AND KENNETH TRAIN, JOURNAL OF APPLIED ECONOMETRICS, J. Appl. Econ. 15: 447-470 (2000).

1. For the very similar Bayesian problem of Dirichlet Process mixture of gaussians, I understand the answer is yes. Ghosal (2013).
2. When I attended some talks on this topic, it seemed progress had mainly been made using KL divergence. See Harry van Zanten's slides.
3. I'm not clear. However, this looks like a source separation problem ($P_N, P_S$ unkown). These are generally much more difficult than mixture modelling alone. In particular for the simple case of $P_N = P_S = N(0,1)$ you wouldn't be able to identify the true $X$ and $Y$ due to symmetry of the distributions about zero.
4. See the fourth of the slides linked above, there's a list of Bayesian models for which convergence guarantees hold.

Say $S_n$ is the class of all Gaussian mixtures with $n$ components. For any continuous distribution $P$ on the reals, are we guaranteed that as $n$ grows, we can approximate $P$ with a GMM with negligible loss in the sense of relative entropy? That is, does $$\lim_{n\rightarrow \infty}\inf_{\hat{P}\in S_n} D(P||\hat{P})=0?$$

No. You can only hope that a KL divergence $D(P\|Q)$ is small if you know that $Q$'s tails are eventually of the same order as $P$'s. This isn't true in general. It is not hard to see that for $P$ Cauchy then for any $n$, $$\inf_{\hat{P}\in S_n} D(P||\hat{P})=\infty$$

More conditions on $P$ are needed to say that.

Say we have a continuous distribution $P$ and we have found an $N$-component Gaussian mixture $\hat{P}$ which is close to $P$ in total variation: $\delta(P,\hat{P})<\varepsilon$. Can we bound $D(P||\hat{P})$ in terms of $\epsilon$?

No. The same example above applies.

If we want to observe $X\sim P_X$ through independent additive noise $Y\sim P_Y$ (both real, continuous), and we have GMMs $\hat{X} \sim Q_X, \hat{Y} \sim Q_Y$ where $\delta(P,Q)<\epsilon$, then is this value small: $$\left|\mathsf{mmse}(X|X+Y)-\mathsf{mmse}(\hat{X}| \hat{X}+\hat{Y})\right|,$$ i.e. is it true that estimating $X$ through $Y$ noise is about as hard as estimating $\hat{X}$ through $\hat{Y}$ noise?

I don't know. If $X,Y,\hat{X},\hat{Y}$ have finite mean and variance then the MMSEs are $E[X|Y]$ and $E[\hat{X}|\hat{Y}]$ (simple derivation here). With these assumptions, the object is to determine whether $|E_P[(E_P[X|Y]-X)^2]-E_Q[(E_Q[X|Y]-X)^2]|$ is small when $TV(P,Q)$ is small. Related.

I haven't been able to prove this, either in general or using the extra additive structure we have assumed on P,Q, or come up with any counterexamples.

Can you do it for non-additive noise models like Poisson noise?

This is ambiguous. In the context of the previous question, if the statement in that answer can be proven in general then the answer is yes.