# Mutual Information from Multiple Sources

The mutual information gain expression is

$$H(X) - H(X | Y)$$

If I have a set of data sources, $\mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace$, then I start with the simplest mutual information analysis I could think of: "Leave One Out" mutual information gain.

So, for each $X_i \in \mathbf{X}$, I calculate

$$I_m = \lbrace H(X_i) - H(\mathbf{X}) : X_i \in \mathbf{X}\rbrace$$

Where $I_m$ indicates "mutual information gain."

However, this seems like it would fail to capture deeper interactions with data sources. Obviously, analyzing all possible combinations is exponential...

But is there a deeper, more robust way of handling "mutual information gain" in this context?

Perhaps a way to sample the various combinations?

• Personally, I can't understand your question. – Ami Tavory Feb 27 '18 at 19:09
• @AmiTavory oh, this is a standard topic in ML. I am using set builder notation in the latter part. – donlan Feb 27 '18 at 19:10
• @AmiTavory for instance, en.wikipedia.org/wiki/Multivariate_mutual_information – donlan Feb 28 '18 at 16:44
• Sorry for the downvote (I'll be happy to reverse it if the question becomes clearer). The problem, IMHO, isn't with standard definitions in information theory or set notations, but with the clarity of the question itself: 1. (minor:) in "mutual information gain equation" I think you meant "expression", not "equation", 2. $I_m = ...$ does not appear to be related to mutual information, as it is a set of differences in entropies, not a set of differences between entropies and conditional entropies, 3. etc. – Ami Tavory Mar 1 '18 at 1:46
• @AmiTavory note the bold font of the second X. Conditional entropy is a difficult concept when dealing with a graph, so it seems like the most logical next step – donlan Mar 1 '18 at 1:48