I am skeptical about the notion that if mutual information between two random variables is non-zero (existence) this doesn't imply the existence of correlation between them BUT existence of correlation does imply the existence of mutual information. i-e

$$ \rho \implies I(X,Y)$$

Is it correct in general?

  • $\begingroup$ Yes, it is correct. Correlation can identify linear (or even monotonic) relations, whereas mutual information is not limited to such types of relations. $\endgroup$
    – George
    Jul 31 '15 at 9:16
  • 5
    $\begingroup$ We must understand your use of the word "existence" as meaning "is nonzero," for otherwise these statements are wrong. (The mutual information always exists but the correlation might not.) This relationship becomes more apparent when stated as the contrapositive: when the mutual information is zero, the correlation must be zero. $\endgroup$
    – whuber
    Jul 31 '15 at 12:15

Mutual information is zero if and only if $p(x,y) = p(x) p(y)$ and this condition implies that correlation is zero. So, if correlation is non-zero, then mutual information need to be non-zero.

  • $\begingroup$ don't you mean "if correlation is zero, mutual information need not be zero"? $\endgroup$
    – develarist
    Oct 28 '20 at 21:53
  • 1
    $\begingroup$ @develarist No, if there is nonzero correlation, there is dependence between the marginal variables, therefore mutual information. "Need not be" means that, while it is not definite, nothing has ruled it out, either. Having nonzero correlation rules out the possibility of zero mutual information. $\endgroup$
    – Dave
    Oct 28 '20 at 22:06
  • $\begingroup$ I think users are more concerned of drawing false conclusions regarding the presence of interrelationships between variables if correlation is found to be zero, which is where mutual information comes to the rescue. Non-zero correlation and non-zero mutual information is meanwhile an obvious corollary $\endgroup$
    – develarist
    Oct 28 '20 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.