When discussing the concept of mixtures of distributions in my machine learning textbook, the authors state the following:

A Gaussian mixture model is a universal approximator of densities, in the sense that any smooth density can be approximated with any specific nonzero amount of error by a Gaussian mixture model with enough components.

I only have a basic background in probability and statistics, and I have absolutely no idea what this section is saying.

I would greatly appreciate it if people could please take the time to explain this in a way that is more understandable to someone of my level.

  • $\begingroup$ I applied this idea once, very early in my career, to carry out quantum-mechanical calculations by using a basis of Gaussians to model bonds in organic molecules. After weeks of running into problems I realized, to my chagrin, that the densities I was trying to approximate were so sharply peaked that it was hopeless to develop a good approximation without using a ridiculously large number of Gaussians, which are too smooth for the job. Thus, one should always be cognizant of the distinction between a purely mathematical theorem and the practical needs of computation. $\endgroup$ – whuber Sep 3 '18 at 13:33

The idea is that an arbitrary density on $\mathbb{R}$, $f(\cdot)$, can be approximated by a Gaussian mixture model $$g_k(\cdot;\boldsymbol{\omega,\mu,\sigma})=\sum_{i=1}^k \omega_i \varphi(\cdot;\mu_i,\sigma_i)$$ in the sense that $$\mathfrak{D}(f(\cdot),g_k(\cdot;\boldsymbol{\omega,\mu,\sigma})\stackrel{k\to\infty}{\longrightarrow}0$$ for some specific functional distance $\mathfrak{D}(\cdot,\cdot)$. This result only applies for weaker types of distance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.