# Understanding Multidimensional Mutual Information

Given random variables $$\vec{x}, \vec{y} \in \mathbb{R}^n$$, and the mutual information, defined as

$$I(\vec{x} : \vec{y}) = H(\vec{x}) + H(\vec{y}) - H(\vec{x}, \vec{y})$$

is it true that

$$I(\vec{x}: \vec{y}) \geq \sum_i I(x_i: y_i)$$

My interpretation is that that collectively several variables should be able to predict another set of variables at least as good as individually. However, when I compute the quantities in this equation from some data I have, I get the opposite, namely $$I(\vec{x}: \vec{y}) < \sum_i I(x_i, y_i)$$. Can one prove the above inequality? Does my code have a bug, or is it the understanding that is wrong? If I am indeed wrong, can you provide another information-theoretic measure that can be used to demonstrate my interpretation.

Edit: I think I can prove mathematically that in fact the opposite is true, but I don't understand why

Edit 2: I have received a satisfactory answer to the first part of the question on the math forum. The remaining question is whether there is a redundancy-corrected version of mutual information, that only measures synergy?

• Are the components in each vector $x$ and $y$ independent ? Mar 3, 2020 at 12:51
• @CamilleGontier You were on the right track, have a look at the link in the edit Mar 3, 2020 at 15:47
• Ok, maybe this paper will be of some help : Rosas, Fernando E., et al. "Quantifying high-order interdependencies via multivariate extensions of the mutual information." Physical Review E 100.3 (2019): 032305. Mar 3, 2020 at 17:41
• @CamilleGontier Thanks, I'll have a read Mar 3, 2020 at 18:37

The intuitional answer is that when you sum mutual informations pairwise, you recount the intersected information inside those variables. Let $$\vec{x}=[X,X],\vec{y}=[Y,Y]$$. Collectively, the intrinsic information is not more than the information between $$X$$ and $$Y$$. But, if you sum them pairwise, you get twice the information, i.e. $$2I(X;Y)$$.