(This question differs from a similarly titled one because mine focuses instead on the analytical solution of mutual information as a function of correlation, its usage, and its seeming pointlessness as a replacement to correlation.)

The closed-form analytical solution for mutual information (a scalar) between two jointly Gaussian distributed random variables $X$ and $Y$ is $$I(X,Y) =-\frac{1}{2}\ln(|\rho|)$$ where $\rho$ is the joint correlation matrix between $X$ and $Y$.

  • If $I(X,Y)$ can pick up non-linear co-dependencies that correlation can't, why do I not see this indicated anywhere in the above formula?
  • If $I(X,Y)$ is just a re-expression of correlation as shown above, what's the point of even switching to mutual information from correlation?
  • Is $I(X,Y)$'s advantage of picking up full co-dependencies that correlation can't only apparent for joint non-Gaussian variables, where the above analytical solution does not apply?
  • 1
    $\begingroup$ Isn't that for when the bivariate distribution is jointly Gaussian? Two Gaussian distributions can be uncorrelated yet most definitely not independent. $\endgroup$
    – Dave
    Oct 28 '20 at 22:12
  • $\begingroup$ thanks for the correction $\endgroup$
    – develarist
    Oct 28 '20 at 22:14

When a bivariate distribution is jointly Gaussian, it means that the dependence structure is correlation.

Therefore, mutual information and correlation magnitude between the marginals become synonyms.

Note, however, that the mutual information does not give you the direction of correlation (nor should it).

  • $\begingroup$ and non-Gaussian joints? what about them $\endgroup$
    – develarist
    Oct 29 '20 at 0:46
  • 2
    $\begingroup$ Then your $\frac{1}{2}\log(\vert\rho\vert)$ formula does not apply. $\endgroup$
    – Dave
    Oct 29 '20 at 1:15
  • $\begingroup$ when the analytical solution no longer applies, then does the non-analytical stand-alone formula for mutual information suddenly have an upper hand over correlation? i.e. is it no longer a function of correlation whatsoever? $\endgroup$
    – develarist
    Oct 29 '20 at 1:18
  • 3
    $\begingroup$ When the bivariate distribution is not jointly Gaussian, then correlation doesn’t tell the whole story. You might be interested in this answer: stats.stackexchange.com/a/30205/247274. There’s a relationship between mutual information and copula. $\endgroup$
    – Dave
    Oct 29 '20 at 1:25
  • 1
    $\begingroup$ When the bivariate distribution is bivariate Gaussian, Pearson correlation is the parameter of the Gaussian copula. Pearson correlation captures all of the dependence structure, so all of the mutual information. Mutual information would be more useful when there is a funkier dependence structure, say an X shape instead of a slash or backslash. $\endgroup$
    – Dave
    Oct 29 '20 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.