I'm working with a high-dimensional mixture distribution, and I'm interested in calculating its entropy.

I think I could work it out if there were only two mixture components. Following @Daniel's suggestion in the comments, here's what I have so far:

If the two components had equal weight, I think the answer would be $\frac{1}{2}(H_1 + H_2 - I_{1,2})$, where $H_1$ and $H_2$ are the entropies of the component distributions and $I_{1,2}$ is the mutual information between them. My logic is that subtracting the mutual information prevents double-counting any entropy.

With a bit more work, I could probably figure out how to generalize it to unequal weights. But my problem involves thousands of mixture components, and I don't know how I'd go about accounting for all of them.

  • 2
    $\begingroup$ Maybe you can write the equations, up the point you have solved? $\endgroup$ – Daniel Dec 2 '13 at 8:15
  • $\begingroup$ @Daniel Good idea. I've added what I have so far. $\endgroup$ – David J. Harris Dec 2 '13 at 8:33
  • $\begingroup$ What to you think about this? $\sum_i H_i - \sum_{i,j} I_{i,j}$ multiplied by some constant $\endgroup$ – Daniel Dec 3 '13 at 0:36
  • $\begingroup$ I don't understand the question (could you refine what is data or information your start with and your goal?), but you might find the Venn diagram on Mutual Information enlightening. $\endgroup$ – Piotr Migdal Feb 19 '14 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.