How does the number of components in a GMM relate to the information content? Say you fit a Gaussian Mixture Model (GMM) to your data using a Bayesian technique, which should tell you the number of components needed to fit your data. Does this also give insight into the information complexity of your data? I'm not sure the correct measure of complexity here, maybe more components <=> higher entropy?
In an intuitive sense, a signal with more features should require more components to describe it. Conversely, a signal that has been low-pass-filtered (reducing high-frequency signal) has less complexity and should require fewer Gaussian components to describe it. Is there a formal relationship between these concepts?
 A: Entropy is generally a reasonable measure for "how much information" a sample of a distribution conveys. So what's the entropy of a mixture of Gaussians?
The entropy of a single Gaussian with covariance matrix $\Sigma$ in $N$ dimensions is $$\frac12 \log (2\pi e)^N |\Sigma|$$ (simplified from Wikipedia as the calculation is a bit tedious). This means that as the eigenvalues of the covariance matrix get bigger (that is, the distribution gets more spread out), the entropy goes up (each data point contains more information). Also note that we can make $|\Sigma|$ as big or small as we like, so the entropy of an individual Gaussian can be any real number.
Unfortunately, calculating the entropy of an arbitrary mixture distribution is more difficult, because you're integrating $\left(\sum c_i p_i\right) \log \left(\sum c_i p_i\right)$, and the logarithm no longer cancels with the exponential in the Gaussian pdf. However, you can get some reasonable intuitions for how the entropy changes with a mixture in special cases.


*

*If you have a single Gaussian, you can approximate it arbitrarily well by a mixture of many more Gaussians (e.g. this is what Gaussian kernel density estimates do). So the entropy of the mixture will be approximately the same as the entropy of the Gaussian that you're trying to approximate. This shows that the entropy does not always increase as you add more components to the mixture.

*More generally, a mixture of sufficiently many Gaussians can approximate any distribution. So you should be able to achieve basically whatever entropy you want to. This also shows that "information required to specify the distribution" and "information contained in a sample of the distribution" aren't necessarily the same.

*There are some analytical upper and lower bounds on the entropy that have been calculated in this paper, but they're pretty nasty.
However, there is an important tractable special case. If the Gaussians are far enough away from each other that they approximately don't overlap, then we actually can compute the entropy of the mixture $p(x) = \sum_{i=1}^M c_ip_i(x)$: it's $$\sum_{i=1}^M -c_i \log c_i + c_i H(p_i).$$
In the extremely special case where your individual Gaussians in the mixture all have the same entropy $H$, then the second term always adds up to $H$ (because the $c_i$ always sum to 1). That means that the entropy of this type of mixture with $m$ elements is just $H + \sum_{i=1}^M -c_i \log c_i$. If the second term looks familiar, that's because it's just the entropy of an $M$-element discrete distribution with weights $c_i$.
So basically, if your Gaussians are far enough apart and identical, the "increase in information" from adding more elements to a mixture behaves the same way as if you were adding those elements to a discrete distribution, which is perhaps intuitive. However, if you're changing the covariance matrix while you add more elements to the mixture, then it depends on how the covariance changes--it could decrease the entropy, if the mixture components get sufficiently "spikier" when you add more of them.
