# Information entropy of a mixture distribution

I'm working with a high-dimensional mixture distribution, and I'm interested in calculating its entropy.

I think I could work it out if there were only two mixture components. Following @Daniel's suggestion in the comments, here's what I have so far:

If the two components had equal weight, I think the answer would be $\frac{1}{2}(H_1 + H_2 - I_{1,2})$, where $H_1$ and $H_2$ are the entropies of the component distributions and $I_{1,2}$ is the mutual information between them. My logic is that subtracting the mutual information prevents double-counting any entropy.

With a bit more work, I could probably figure out how to generalize it to unequal weights. But my problem involves thousands of mixture components, and I don't know how I'd go about accounting for all of them.

• Maybe you can write the equations, up the point you have solved? – Daniel Dec 2 '13 at 8:15
• @Daniel Good idea. I've added what I have so far. – David J. Harris Dec 2 '13 at 8:33
• What to you think about this? $\sum_i H_i - \sum_{i,j} I_{i,j}$ multiplied by some constant – Daniel Dec 3 '13 at 0:36
• I don't understand the question (could you refine what is data or information your start with and your goal?), but you might find the Venn diagram on Mutual Information enlightening. – Piotr Migdal Feb 19 '14 at 23:22