I need to estimate as fast and accurately as possible the differential entropy of a mixture of $K$ multivariate Gaussians:

$$ \mathcal{H}[q] = -\sum_{k=1}^K w_k \int q_k(\textbf{x}) \log \left[\sum_{j=1}^K w_j q_j(\textbf{x}) \right] d\textbf{x}, $$

where $q_k$ are individual multivariate normals and $w_k$ are the mixture weights. In general, there is no analytical solution.

Assumptions: We can assume that all mixture components have diagonal covariance matrices. If it helps, I could also assume that all mixture weights are equal, $w_k \equiv 1/K$. The dimensionality goes say from 1 to 20.

In the literature I found some analytical lower/upper bounds for $\mathcal{H}$, and an iterative method that computes $\mathcal{H}$ by splitting the Gaussian components [1]. The lower bound has been used e.g. in [2]. See also this previous question.

Alternatively, the entropy could be computed by Monte Carlo methods. For example, it is easy to sample from each $q_k$ in turn, and evaluate the logarithm (simple Monte Carlo). However, there could be better ways of doing it via variance reduction techniques.

Any suggestion or reference that could be useful for a (smarter) Monte Carlo approach?


  1. Huber, M. F., Bailey, T., Durrant-Whyte, H., & Hanebeck, U. D. (2008, August). On entropy approximation for Gaussian mixture random vectors. In Multisensor Fusion and Integration for Intelligent Systems, 2008. MFI 2008. IEEE International Conference on (pp. 181-188). IEEE.
  2. Gershman, S. J., Hoffman, M. D., & Blei, D. M. (2012, June). Nonparametric variational inference. In Proceedings of the 29th International Coference on International Conference on Machine Learning (pp. 235-242). Omnipress.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.