What are the probability density functions that GMM can approximate? a reference in appreciated about this.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Goodfellow et al. 2016, p. 65 states:
A Gaussian mixture model is a universal approximator of densities, in the sense that any smooth density can be approximated with any specific nonzero amount of error by a Gaussian mixture model with enough components.
M. Carreira ascribes this property to kernel density estimation with reference to Scott 1992 and another source that I could not find. Given the connection between KDE and GMMs this is understandable.
The user Xi'an provided an explanation for the above statement in this answer.
While this answers the question, it has to be noted that while it can theoretically approximate any smooth density, it shouldn't be used as a general purpose model.
Fitting a mixture of many components can quickly become more computationally expensive than using a more fitting parametric model. Examples of this could be distributions with very thin peaks, which one needs to approximate with very thin bandwidths, as well as distributions with long tails, which will be difficult to get right with either very wide Gaussians or many small ones.
In these and probably many other cases it is preferrable to use a more fitting, if less general, model.
$\begingroup$
$\endgroup$
Dalal and Hall's "Approximating Priors by Mixtures of Natural Conjugate Priors" discusses estimating arbitrary densities with mixtures of normals (or, for that matter, mixtures of other conjugate prior densities). This paper and papers which cite it discuss the details of how to approximate densities arbitrarily well with GMMs and other mixtures. This can be a useful way to approximate priors, since a mixture of conjugate priors is also conjugate.
Dalal, S. R., and W. J. Hall. "Approximating Priors by Mixtures of Natural Conjugate Priors." Journal of the Royal Statistical Society. Series B (Methodological), vol. 45, no. 2, 1983, pp. 278–286. JSTOR, www.jstor.org/stable/2345533. Accessed 7 Dec. 2020.
