7
$\begingroup$

(Let $X$ and $Y$ be random variables, sufficiently nice for my question to make sense.)

$$ \text{Correlation} $$

$$ \rho(X, Y) = \dfrac{\text{cov}(X, Y)}{\sqrt{\text{var}(X)}\sqrt{\text{var}(Y)}} $$

If we draw analogies between information theory and classical statistics, entropy is analogous to variance, and mutual information is analogous to covariance. Running with that idea, I should be able to make some kind of "information" correlation, $\rho_I$.

$$ \rho_I(X, Y) = \dfrac{\text{MI}(X, Y)}{\sqrt{\text{MI}(X, X)}\sqrt{\text{MI}(Y, Y)}} = \dfrac{\text{MI}(X, Y)}{\sqrt{\text{H}(X)}\sqrt{\text{H}(Y)}} $$

Does this make sense? Has any literature explored this idea? I am working on a project where it would be convenient to be able to say that some percentage of the entropy in $Y$ is explained by $X$ (something like the usual interpretation of $R^2$ for variance). Better yet, I would like to be able to say that $X$ contains some percentage of the "information" contained in $Y$.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ The division by standard deviations in the first formula is the result of an analysis, not a definition. That analysis begins by standardizing $X$ and $Y$ and then invoking the bilinear property of covariance. Because MI is not bilinear, you cannot even get started with that program. That leaves you only with a suggestive but meaningless formula. $\endgroup$
    – whuber ♦
    Commented Dec 28, 2020 at 13:56
  • $\begingroup$ @whuber What you say about first standardizing makes sense. However, what I’ve posted appears to be in use: Strehl, Alexander, and Joydeep Ghosh. "Cluster ensembles---a knowledge reuse framework for combining multiple partitions." Journal of machine learning research 3.Dec (2002): 583-617. jmlr.org/papers/volume3/strehl02a/strehl02a.pdf And it behaves about the way I’d expect as far as saying that some percentage of information is contained in one variable compared to the other. Have you ever seen Strehl’s paper? $\endgroup$
    – Dave
    Commented Dec 29, 2020 at 23:51
  • $\begingroup$ I am not aware of that Dave, and thank you for bringing it to my attention. $\endgroup$
    – whuber ♦
    Commented Dec 30, 2020 at 0:05

2 Answers 2

Reset to default
6
$\begingroup$

Taking a different angle from the information coefficient of correlation, I'll give background on the formula you gave. It's called the normalized mutual information.

Unfortunately, lots of things are called the normalized mutual information. What you've shown uses the geometric mean of $H(X)$ and $H(Y)$ as the denominator. Other common choices are the arithmetic mean, the min, and the max. Any generalized mean will do. Another choice of denominator is the joint entropy $H(X, Y)$.

Because of this, it's important to give the formula you use for normalized mutual information.

That being said, there are big problems with the normalized mutual information. It suffers from what's called the finite size effect; the baseline keeps steadily rising, and you can get a high NMI in cases when one of the variables conveys nothing. It's better to use an adjusted-for-chance variant ("adjusted mutual information") whose expectation (rather than its minimum) is 0.

Improve this answer
$\endgroup$
5
$\begingroup$

To answer your question "Has any literature explored this idea?". Linfoot (1957) introduced the informational coefficient of correlation, $\mathit{(IC)}$:

$$IC=\sqrt{1-e^{-2\cdot{I(X;Y)}}}$$

where $\mathit{I(X;Y)}$ is the mutual information.

While it does not seem to be a commonly used statistic in literature, in may satisfy your requirements. (Note: $\mathit{IC}$ is a measure of ${r}$, not ${r^2}$).

Reference: Linfoot, E.H. (1957). An informational measure of correlation. Information and Control, 1, 85-89.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.