6
$\begingroup$

I need to estimate as fast and accurately as possible the differential entropy of a mixture of $K$ multivariate Gaussians:

$$ \mathcal{H}[q] = -\sum_{k=1}^K w_k \int q_k(\textbf{x}) \log \left[\sum_{j=1}^K w_j q_j(\textbf{x}) \right] d\textbf{x}, $$

where $q_k$ are individual multivariate normals and $w_k$ are the mixture weights. In general, there is no analytical solution.

Assumptions: We can assume that all mixture components have diagonal covariance matrices. If it helps, I could also assume that all mixture weights are equal, $w_k \equiv 1/K$. The dimensionality goes say from 1 to 20.

In the literature I found some analytical lower/upper bounds for $\mathcal{H}$, and an iterative method that computes $\mathcal{H}$ by splitting the Gaussian components [1]. The lower bound has been used e.g. in [2]. See also this previous question.

Alternatively, the entropy could be computed by Monte Carlo methods. For example, it is easy to sample from each $q_k$ in turn, and evaluate the logarithm (simple Monte Carlo). However, there could be better ways of doing it via variance reduction techniques.

Any suggestion or reference that could be useful for a (smarter) Monte Carlo approach?

References:

  1. Huber, M. F., Bailey, T., Durrant-Whyte, H., & Hanebeck, U. D. (2008, August). On entropy approximation for Gaussian mixture random vectors. In Multisensor Fusion and Integration for Intelligent Systems, 2008. MFI 2008. IEEE International Conference on (pp. 181-188). IEEE.
  2. Gershman, S. J., Hoffman, M. D., & Blei, D. M. (2012, June). Nonparametric variational inference. In Proceedings of the 29th International Coference on International Conference on Machine Learning (pp. 235-242). Omnipress.
$\endgroup$
5
  • $\begingroup$ Did you figure this out? $\endgroup$
    – user3180
    May 8, 2021 at 8:10
  • $\begingroup$ I tried many things including some variance reduction techniques but nothing worked better (in general) than simple Monte Carlo. $\endgroup$
    – lacerbi
    May 9, 2021 at 9:03
  • $\begingroup$ So no importance sampling? I think I googled some importance sampling thing $\endgroup$
    – user3180
    May 9, 2021 at 10:51
  • $\begingroup$ Estimating Mixture Entropy with Pairwise Distances is another link.. $\endgroup$
    – user3180
    May 9, 2021 at 11:03
  • $\begingroup$ I tried importance sampling, and it did not work particularly well (no better than simple Monte Carlo). I read "Estimating Mixture Entropy with Pairwise Distances" at the time and it only has lower/upper bounds, there is nothing to be used there (if the goal is to get a good unbiased estimate). $\endgroup$
    – lacerbi
    May 10, 2021 at 11:09

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.