Given random variables $\vec{x}, \vec{y} \in \mathbb{R}^n$, and the mutual information, defined as

$I(\vec{x} : \vec{y}) = H(\vec{x}) + H(\vec{y}) - H(\vec{x}, \vec{y})$

is it true that

$I(\vec{x}: \vec{y}) \geq \sum_i I(x_i: y_i)$

My interpretation is that that collectively several variables should be able to predict another set of variables at least as good as individually. However, when I compute the quantities in this equation from some data I have, I get the opposite, namely $I(\vec{x}: \vec{y}) < \sum_i I(x_i, y_i)$. Can one prove the above inequality? Does my code have a bug, or is it the understanding that is wrong? If I am indeed wrong, can you provide another information-theoretic measure that can be used to demonstrate my interpretation.

Edit: I think I can prove mathematically that in fact the opposite is true, but I don't understand why

Edit 2: I have received a satisfactory answer to the first part of the question on the math forum. The remaining question is whether there is a redundancy-corrected version of mutual information, that only measures synergy?

  • $\begingroup$ Are the components in each vector $x$ and $y$ independent ? $\endgroup$ Mar 3, 2020 at 12:51
  • 1
    $\begingroup$ @CamilleGontier You were on the right track, have a look at the link in the edit $\endgroup$ Mar 3, 2020 at 15:47
  • 1
    $\begingroup$ Ok, maybe this paper will be of some help : Rosas, Fernando E., et al. "Quantifying high-order interdependencies via multivariate extensions of the mutual information." Physical Review E 100.3 (2019): 032305. $\endgroup$ Mar 3, 2020 at 17:41
  • 1
    $\begingroup$ @CamilleGontier Thanks, I'll have a read $\endgroup$ Mar 3, 2020 at 18:37

1 Answer 1


The intuitional answer is that when you sum mutual informations pairwise, you recount the intersected information inside those variables. Let $\vec{x}=[X,X],\vec{y}=[Y,Y]$. Collectively, the intrinsic information is not more than the information between $X$ and $Y$. But, if you sum them pairwise, you get twice the information, i.e. $2I(X;Y)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.