References that justify use of Gaussian Mixtures

Question

Gaussian mixture models (GMMs) are appealing because they are simple to work with both in analytically and in practice, and are capable of modeling some exotic distributions without too much complexity. There are a few analytic properties we should expect to hold which are not clear in general. In particular:

• Say $S_n$ is the class of all Gaussian mixtures with $n$ components. For any continuous distribution $P$ on the reals, are we guaranteed that as $n$ grows, we can approximate $P$ with a GMM with negligible loss in the sense of relative entropy? That is, does $$\lim_{n\rightarrow \infty}\inf_{\hat{P}\in S_n} D(P||\hat{P})=0?$$
• Say we have a continuous distribution $P$ and we have found an $N$-component Gaussian mixture $\hat{P}$ which is close to $P$ in total variation: $\delta(P,\hat{P})<\varepsilon$. Can we bound $D(P||\hat{P})$ in terms of $\epsilon$?
• If we want to observe $X\sim P_X$ through independent additive noise $Y\sim P_Y$ (both real, continuous), and we have GMMs $\hat{X} \sim Q_X, \hat{Y} \sim Q_N$ where $\delta(P,Q)<\epsilon$, then is this value small: $$\left|\mathsf{mmse}(X|X+Y)-\mathsf{mmse}(\hat{X}| \hat{X}+\hat{Y})\right|,$$ i.e. is it true that estimating $X$ through $Y$ noise is about as hard as estimating $\hat{X}$ through $\hat{Y}$ noise?
• Can you do it for non-additive noise models like Poisson noise?

My (short) literature review so far has just turned up very applied tutorials. Does anyone have any references that rigorously demonstrate under what conditions we are justified in using mixture models?

Answer 1

With respect to your questions:

1. For the very similar Bayesian problem of Dirichlet Process mixture of gaussians, I understand the answer is yes. Ghosal (2013).
2. When I attended some talks on this topic, it seemed progress had mainly been made using KL divergence. See Harry van Zanten's slides.
3. I'm not clear. However, this looks like a source separation problem ($P_N, P_S$ unkown). These are generally much more difficult than mixture modelling alone. In particular for the simple case of $P_N = P_S = N(0,1)$ you wouldn't be able to identify the true $X$ and $Y$ due to symmetry of the distributions about zero.
4. See the fourth of the slides linked above, there's a list of Bayesian models for which convergence guarantees hold.

Answer 2

In econometrics, where the context is of mixture distributions of coefficients in logit models, the standard reference is: MIXED MNL MODELS FOR DISCRETE RESPONSE DANIEL MCFADDEN AND KENNETH TRAIN, JOURNAL OF APPLIED ECONOMETRICS, J. Appl. Econ. 15: 447-470 (2000).

Answer 3

Here is a partial answer.

Say $S_n$ is the class of all Gaussian mixtures with $n$ components. For any continuous distribution $P$ on the reals, are we guaranteed that as $n$ grows, we can approximate $P$ with a GMM with negligible loss in the sense of relative entropy? That is, does $$\lim_{n\rightarrow \infty}\inf_{\hat{P}\in S_n} D(P||\hat{P})=0?$$

No. You can only hope that a KL divergence $D(P\|Q)$ is small if you know that $Q$'s tails are eventually of the same order as $P$'s. This isn't true in general. It is not hard to see that for $P$ Cauchy then for any $n$, $$\inf_{\hat{P}\in S_n} D(P||\hat{P})=\infty$$

More conditions on $P$ are needed to say that.

Say we have a continuous distribution $P$ and we have found an $N$-component Gaussian mixture $\hat{P}$ which is close to $P$ in total variation: $\delta(P,\hat{P})<\varepsilon$. Can we bound $D(P||\hat{P})$ in terms of $\epsilon$?

No. The same example above applies.

If we want to observe $X\sim P_X$ through independent additive noise $Y\sim P_Y$ (both real, continuous), and we have GMMs $\hat{X} \sim Q_X, \hat{Y} \sim Q_Y$ where $\delta(P,Q)<\epsilon$, then is this value small: $$\left|\mathsf{mmse}(X|X+Y)-\mathsf{mmse}(\hat{X}| \hat{X}+\hat{Y})\right|,$$ i.e. is it true that estimating $X$ through $Y$ noise is about as hard as estimating $\hat{X}$ through $\hat{Y}$ noise?

I don't know. If $X,Y,\hat{X},\hat{Y}$ have finite mean and variance then the MMSEs are $E[X|Y]$ and $E[\hat{X}|\hat{Y}]$ (simple derivation here). With these assumptions, the object is to determine whether $|E_P[(E_P[X|Y]-X)^2]-E_Q[(E_Q[X|Y]-X)^2]|$ is small when $TV(P,Q)$ is small. Related.

I haven't been able to prove this, either in general or using the extra additive structure we have assumed on P,Q, or come up with any counterexamples.

Can you do it for non-additive noise models like Poisson noise?

This is ambiguous. In the context of the previous question, if the statement in that answer can be proven in general then the answer is yes.