103
$\begingroup$

Why and when we should use Mutual Information over statistical correlation measurements such as "Pearson", "spearman", or "Kendall's tau" ?

$\endgroup$

7 Answers 7

143
$\begingroup$

Let's consider one fundamental concept of (linear) correlation, covariance (which is Pearson's correlation coefficient "un-standardized"). For two discrete random variables $X$ and $Y$ with probability mass functions $p(x)$, $p(y)$ and joint pmf $p(x,y)$ we have

$$\operatorname{Cov}(X,Y) = E(XY) - E(X)E(Y) = \sum_{x,y}p(x,y)xy - \left(\sum_xp(x)x\right)\cdot \left(\sum_yp(y)y\right)$$

$$\Rightarrow \operatorname{Cov}(X,Y) = \sum_{x,y}\left[p(x,y)-p(x)p(y)\right]xy$$

The Mutual Information between the two is defined as

$$I(X,Y) = E\left (\ln \frac{p(x,y)}{p(x)p(y)}\right)=\sum_{x,y}p(x,y)\left[\ln p(x,y)-\ln p(x)p(y)\right]$$

Compare the two: each contains a point-wise "measure" of "the distance of the two rv's from independence" as it is expressed by the distance of the joint pmf from the product of the marginal pmf's: the $\operatorname{Cov}(X,Y)$ has it as difference of levels, while $I(X,Y)$ has it as difference of logarithms.

And what do these measures do? In $\operatorname{Cov}(X,Y)$ they create a weighted sum of the product of the two random variables. In $I(X,Y)$ they create a weighted sum of their joint probabilities.

So with $\operatorname{Cov}(X,Y)$ we look at what non-independence does to their product, while in $I(X,Y)$ we look at what non-independence does to their joint probability distribution.

Reversely, $I(X,Y)$ is the average value of the logarithmic measure of distance from independence, while $\operatorname{Cov}(X,Y)$ is the weighted value of the levels-measure of distance from independence, weighted by the product of the two rv's.

So the two are not antagonistic—they are complementary, describing different aspects of the association between two random variables. One could comment that Mutual Information "is not concerned" whether the association is linear or not, while Covariance may be zero and the variables may still be stochastically dependent. On the other hand, Covariance can be calculated directly from a data sample without the need to actually know the probability distributions involved (since it is an expression involving moments of the distribution), while Mutual Information requires knowledge of the distributions, whose estimation, if unknown, is a much more delicate and uncertain work compared to the estimation of Covariance.

$\endgroup$
15
  • 1
    $\begingroup$ @ Alecos Papadopoulos; Thanks for your comprehensive answer. $\endgroup$
    – SaZa
    Commented Jan 9, 2014 at 9:25
  • 2
    $\begingroup$ I was asking myself the same question but I have not completely understood the answer. @ Alecos Papadopoulos: I understood that the dependance measured is not the same, okay. So for what kind of relations beetween X and Y should we prefer mutual information I(X,Y) rather than Cov(X,Y)? I had a strange example recently where Y was almost linearly dependent on X (it was nearly a straight line in a scatter plot) and Corr(X,Y) was equal to 0.87 whereas I(X,Y) was equal to 0.45. So is there clearly some cases where one indicator should be choosen over the other one? Thanks for helping! $\endgroup$
    – Gandhi91
    Commented Mar 25, 2014 at 19:47
  • 2
    $\begingroup$ This is a great and very clear answer. I was wondering if you have a readily available example where cov is 0, but pmi is not. $\endgroup$
    – thang
    Commented Jun 11, 2017 at 23:54
  • 1
    $\begingroup$ @GENIVI-LEARNER In the context of my answer "point-wise" means that deviations are measured point by point (i.e. value per value that the rv takes). And no, it is not deviation from the mean here, this is clearly stated in my answer. It is deviation from the Independence relation. $\endgroup$ Commented Feb 18, 2020 at 19:39
  • 2
    $\begingroup$ @Cybernetic "The mean is well defined only for the Gaussian"? Last time I checked, there is a really large number of distributions that have a finite mean. And as I already wrote, an otherwise inconsequential truncation allows us to have fat-tail/heavy-tail/long-tail/subexponential distributions together with finite moments (if they are missing in the first place). I understand your criticism for those statisticians that still cling myopically to the "Normal world", but this is a phenomenon of professional inertia. Finally, as my contribution to bold claims, nobody understands probability. $\endgroup$ Commented Feb 19, 2020 at 17:47
16
$\begingroup$

Here's an example.

In these two plots the correlation coefficient is zero. But we can get high shared mutual information even when the correlation is zero.

In the first, I see that if I have a high or low value of X then I'm likely to get a high value of Y. But if the value of X is moderate then I have a low value of Y. The first plot holds information about the mutual information shared by X and Y. In the second plot, X tells me nothing about Y.

Mutual Information vs Correlation

$\endgroup$
12
$\begingroup$

Mutual information is a distance between two probability distributions. Correlation is a linear distance between two random variables.

You can have a mutual information between any two probabilities defined for a set of symbols, while you cannot have a correlation between symbols that cannot naturally be mapped into a R^N space.

On the other hand, the mutual information does not make assumptions about some properties of the variables... If you are working with variables that are smooth, correlation may tell you more about them; for instance if their relationship is monotonic.

If you have some prior information, then you may be able to switch from one to another; in medical records you can map the symbols "has genotype A" as 1 and "does not have genotype A" into 0 and 1 values and see if this has some form of correlation with one sickness or another. Similarly, you can take a variable that is continuous (ex: salary), convert it into discrete categories and compute the mutual information between those categories and another set of symbols.

$\endgroup$
2
  • 1
    $\begingroup$ Correlation isn't a linear function. Should it say that correlation is a measure of the linear relationship between random variables? $\endgroup$ Commented Oct 20, 2016 at 9:33
  • 2
    $\begingroup$ I think this: "You can have a mutual information between any two probabilities defined for a set of symbols, while you cannot have a correlation between symbols that cannot naturally be mapped into a R^N space" is probably the key. Corr doesn't make sense if you don't have a complete random variable; however, pmi makes sense even with just the pdf and sigma (the space). This is why in many applications where RVs don't make sense (e.g. NLP), pmi is used. $\endgroup$
    – thang
    Commented Jun 12, 2017 at 0:02
7
$\begingroup$

Although both of them are a measure of relationship between features, the MI is more general than correlation coefficient (CE) sine the CE is only able to takes into account linear relationships but the MI can also handle non-linear relationships.

$\endgroup$
1
  • 1
    $\begingroup$ @meow Calculating Pearson's correlation does not assume linearity, but rather quantifies the tightness to a line. Spearman's correlation likewise quantifies monotonicity, it does not assume it. Second moments are assumed to exist, but calculating a Pearson's correlation does not assume a joint normal distribution. $\endgroup$
    – Galen
    Commented Sep 8, 2022 at 14:18
2
$\begingroup$

Mutual Information (MI) uses the concept entropy to specify how much common certainty are there in two data samples $X$ and $Y$ with distribution functions $p_{x}(x)$ and $p_y(y)$. Considering this interpretation of MI: $$I(X:Y) = H(X) + H(Y) - H(X,Y)$$ we see that the last part says about the dependency of variables. In case of independence the MI is zero and in case of a consistency between $X$ and $Y$ the MI is equal with the entropy of $X$ or $Y$. Though, the covariance measures only the distance of every data sample $(x,y)$ from the average ($\mu_X, \mu_Y)$. Therefore, Cov is only one part of MI. Another difference is the extra information that Cov can deliver about the sign of Cov. This type of knowledge can not extracted from MI because of log-function.

$\endgroup$
0
$\begingroup$

Note that Correlation(Pearson, Spearman or Kendell) takes values in $[-1,1]$ while Mutual Information takes value in $\mathbb{R^*}$. This makes a big difference: a correlation score is a stronger description of the association between the two RVs than the mutual information. On the other hand, although mutual information is a weaker description it can capture a more general association between two RVs.

A non-zero correlation score not only tells you that the two RVs are related but also in which direction are they related in the value space. This could be very useful in some situations.

Consider the two RVs: the weight and height of a human. If I am told that they are positively correlated, then I can guess the height of a person based on his weight, i.e., a heavy person is likely to be tall(n.b. correlation is not equal to causation).

However, mutual information only tells you if there is some relationship between the two RVs, without precising how exactly the two RVs are related in values. In the same example as above, if I am told that weight and height has a non zero mutual information, then all I know is weight is somehow related to height but I don't know how exactly are they related. It could be negatively related or positively related. (Indeed, an example is a correlation of -1 and +1 both yield maximum mutual information, so the two scenarios are indistinguishable solely based on MI!)

However, the advantage of MI is that it could capture more general(or subtle) associations. Pearson, Spearman, and Kendall are all trying to capture a sort of an increasing or decreasing relationship between RVs in value space. But if the association between the two RVs is quadratic such as shown in the last figure of this reply: then neither of them would capture this relationship and gives 0 as a correlation.

enter image description here

The choice of MI versus a kind of correlation coefficient really depends on what you want to capture. If you want to investigate if there is any sort of association between the two RVs, then choose MI. If you want to find out if the values of two RVs are mutually indicative, then choose a correlation coefficient.

$\endgroup$
0
$\begingroup$
  1. Complex Relationships: When you suspect that the relationship between variables is complex and not just linear or monotonic, MI can provide a more comprehensive measure of dependency.

  2. Mixed Data Types: If you're dealing with a mix of continuous, ordinal, and nominal variables, MI can offer a consistent measure across variable types.

  3. No Assumptions: If you want a measure that doesn't make underlying assumptions about data distributions, MI can be more suitable.

  4. Feature Selection: In machine learning, MI is often used for feature selection since it can identify any statistical dependency between features and targets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.