1
$\begingroup$

Short:

${\bf X} \sim N({\bf 0},{\bf I}+{\bf I}_j)$; ${\bf I}_j\in S=\{I_j: I_j$ is diagonal and $ I_j \succeq 0\}, |S|=K$, and $j\sim U(1,K)$.

What is $h({\bf X})$? What happens when $K\rightarrow\infty$?

Long:

What is the entropy of a multivariate Gaussian mixture, when the mean of each of the component Gaussian distributions is zero and the variance is $\bf I$+$\bf I_j$? Where $\bf I_j$ is from a known set of positive semi-definite diagonal matrices, distributed uniformly.

If the exact entropy cannot be computed, is there a simple upper or lower bound to this value other than using Gaussian approximation and Jensen's inequality, respectively?

Will the computation of this entropy simplify when the cardinality of the set of $\bf I_j$ is very large? Does such an infinite Gaussian mixture converge to any particular distribution?

$\endgroup$
  • $\begingroup$ The entropy cannot be computed in closed form. $\endgroup$ – Xi'an Mar 23 '15 at 12:19
  • $\begingroup$ Thanks @Xi'an . Is it possible to obtain any bounds or approximations? Is there any non-mixture distribution that has the least KL divergence from Gaussian mixture? $\endgroup$ – tln Mar 24 '15 at 9:42
  • $\begingroup$ See mdpi.com/1099-4300/19/7/361/htm for lower and upper bounds (with computation time O(n^2) in the number of components) $\endgroup$ – Brendan Tracey Apr 23 '18 at 21:45
5
$\begingroup$

I don't think there is a closed form representation of the entropy of Gaussian mixture model. However, there are articles on the approximation of the entropy, such as [Huber2008]. Therefore, you might need some approximation methods. I have several thoughts listed below by relevance (that I think) in decreasing order:

  1. The concave property of differential entropy can be exploited. Assuming the Gaussian mixture model has the following density function $$p(\mathbf{x}) = \sum_{i=1}^{m} w_i g(\mathbf{x}|\mathbf{0},\mathbf{I}+\mathbf{I}_j) $$ Then it is true that $$h(\mathbf{X}) = h\left[\sum_{i=1}^{m} w_i g(\mathbf{x}|\mathbf{0},\mathbf{I}+\mathbf{I}_j)\right] \geq \sum_{i=1}^{m} w_i h\left[g(\mathbf{x}|\mathbf{0},\mathbf{I}+\mathbf{I}_j)\right]$$ Now, let's exam every component in the summation, $$ h\left[g(\mathbf{x}|\mathbf{0},\mathbf{I}+\mathbf{I}_j)\right] = \frac{1}{2} \log |\mathbf{I}+\mathbf{I}_j|+\frac{n}{2}\log (2\pi e)$$ where $n$ is the dimension of $\mathbf{X}$. From the property of determinant, it follows that $$\log |\mathbf{I}+\mathbf{I}_j| \geq \text{tr}[\mathbf{I}-(\mathbf{I}+\mathbf{I}_j)^{-1}]$$ which I think can be calculated easily given $\mathbf{I}_j$ are digonal matrices. Therefore, there is a lower bound on $h(\mathbf{X})$, such that $$h(\mathbf{X}) \geq \frac{n}{2}\log(2 \pi e)+\frac{1}{2}\sum_{i=1}^{m}\text{tr}[\mathbf{I}-(\mathbf{I}+\mathbf{I}_j)^{-1}]$$
  2. Another way to deal with your question is to get a non-mixture distribution that has the least divergence from Gaussian mixture as you suggested. I don't think the KL-divergence can be calculated in closed form for Gaussian mixtures, however, you might be able to get an upper bound on the KL-divergence: $$D\left[\sum_{i=1}^{m} w_i g(\mathbf{0},\mathbf{I}+\mathbf{I}_j) \|g(\mathbf{0},\boldsymbol{\Sigma})\right] \leq \sum_{i=1}^{m} w_i D\left[ g(\mathbf{0},\mathbf{I}+\mathbf{I}_j) \|g(\mathbf{0},\boldsymbol{\Sigma})\right]$$ where every component on the right hand side has a closed form representation. Therefore, you might be able to calculate an optimal multivariate Gaussian distribution that has the least distance from the upper bound of the KL divergence.
  3. Although the KL-divergence does not have a closed form for Gaussian mixture models, the Cauchy-Schwarz divergence has a closed form representation for two Gaussian mixture models [Kampa2011]. You can make one of them a multivariate Gaussian distribution, and obtain the optimal covariance matrix by minimizing the Cauchy-Schwarz divergence.

I hope some of the ideas is helpful.

Reference:

Huber, Marco F., et al. "On entropy approximation for Gaussian mixture random vectors." Multisensor Fusion and Integration for Intelligent Systems, 2008. MFI 2008. IEEE International Conference on. IEEE, 2008.

Kampa, Kittipat, Erion Hasanbelliu, and Jose C. Principe. "Closed-form Cauchy-Schwarz PDF divergence for mixture of Gaussians." Neural Networks (IJCNN), The 2011 International Joint Conference on. IEEE, 2011.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.