I was going through the posts that describe the difference between covariance and MI and came across following from Quora

The covariance of two random variables measures the strength of the linear relationship between them, but it's possible for it to be undefined if either random variable doesn't have a well-defined mean.

mutual information between two random variables is always defined, and it measures how much information either carries about the other. The reason that we don't use it widely is simply that it's difficult to estimate.

So my questions are:

  1. How is covariance measuring "linear relationship"? The formula $\operatorname{Cov}(X,Y) = E(XY) - E(X)E(Y)$ seems to only measure the deviation from full independence of rv's in the distribution. So where is the concept of linear relation?
  2. Also what exactly does it mean that mutual information measures how much information one rv carries about the other. So isnt it essentially same as correlation where knowing an increase in value of one rv tells whether the value of other rv will also increase if correlation is positive. So technically isnt that one rv providing an information of the other rv?

Formula for MI copy/pasted from this SE post $I(X,Y) = E\left (\ln \frac{p(x,y)}{p(x)p(y)}\right)=\sum_{x,y}p(x,y)\left[\ln p(x,y)-\ln p(x)p(y)\right]$


1 Answer 1


You can interpret the correlation as a measure of how well a straight line (linear relationship) fits the data. If you have a correlation of $\pm 1,$ then it is a perfect linear relationship. But what if the relationship is not linear? Here's an example: $$\operatorname{kinetic energy}=\frac{\operatorname{mass}\cdot\operatorname{velocity}^2}{2}.$$ A correlation between kinetic energy and velocity, particularly over a large interval, would likely be rather close to zero. However, the mutual information is quite large. So the concept of mutual information is more general, and can capture non-linear relationships, whereas correlation is merely a measure of mutual information if the relationship is linear.

I say that. You can introduce a new feature, call it square-of-velocity, and then do a correlation with that new feature. The new relationship would be linear, and correlation would once again be a good way to measure the mutual information. Here's a more difficult one:

$$Y=A\cos(\omega t+\theta). $$

Now the relationship between $Y$ and the phase angle $\theta$ is non-linear, and you cannot regain linearity by the definition of a new feature.

Incidentally, I would encourage you to investigate Judea Pearl's concepts of causality in his book The Book of Why, and others. It will elucidate some of these ideas very clearly.

  • 1
    $\begingroup$ Awesome explanation with concrete formulation. So will mutual information will always be able to measure the information say in your formula $Y=A\cos(\omega t+\theta)$? What exactly in formulation of MI makes it possible? $\endgroup$ Mar 19, 2020 at 14:51
  • $\begingroup$ Ill update my question to include formulation for MI because I want to understand it intuitively. $\endgroup$ Mar 19, 2020 at 14:51
  • 2
    $\begingroup$ You can calculate a sample covariance with some data. That is not sufficient information to calculate the MI. You need the probability distribution of the RVs in order to do that. $\endgroup$ Mar 19, 2020 at 19:13
  • 1
    $\begingroup$ I think I got it. So I just wanna paraphrase. So you are saying that covariance can be calculated by a "single realization" of the RV in other-words single outcome. MI on the other hand, requires "all realizations" of RV, in other words its distribution. Right? $\endgroup$ Mar 19, 2020 at 20:23
  • 1
    $\begingroup$ Yes, I would agree with that. $\endgroup$ Mar 20, 2020 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.