Difference between Covariance and Mutual Information

Question

I was going through the posts that describe the difference between covariance and MI and came across following from Quora

The covariance of two random variables measures the strength of the linear relationship between them, but it's possible for it to be undefined if either random variable doesn't have a well-defined mean.

mutual information between two random variables is always defined, and it measures how much information either carries about the other. The reason that we don't use it widely is simply that it's difficult to estimate.

So my questions are:

1. How is covariance measuring "linear relationship"? The formula $\operatorname{Cov}(X,Y) = E(XY) - E(X)E(Y)$ seems to only measure the deviation from full independence of rv's in the distribution. So where is the concept of linear relation?
2. Also what exactly does it mean that mutual information measures how much information one rv carries about the other. So isnt it essentially same as correlation where knowing an increase in value of one rv tells whether the value of other rv will also increase if correlation is positive. So technically isnt that one rv providing an information of the other rv?

Formula for MI copy/pasted from this SE post $I(X,Y) = E\left (\ln \frac{p(x,y)}{p(x)p(y)}\right)=\sum_{x,y}p(x,y)\left[\ln p(x,y)-\ln p(x)p(y)\right]$

Answer 1

You can interpret the correlation as a measure of how well a straight line (linear relationship) fits the data. If you have a correlation of $\pm 1,$ then it is a perfect linear relationship. But what if the relationship is not linear? Here's an example: $$\operatorname{kinetic energy}=\frac{\operatorname{mass}\cdot\operatorname{velocity}^2}{2}.$$ A correlation between kinetic energy and velocity, particularly over a large interval, would likely be rather close to zero. However, the mutual information is quite large. So the concept of mutual information is more general, and can capture non-linear relationships, whereas correlation is merely a measure of mutual information if the relationship is linear.

I say that. You can introduce a new feature, call it square-of-velocity, and then do a correlation with that new feature. The new relationship would be linear, and correlation would once again be a good way to measure the mutual information. Here's a more difficult one:

$$Y=A\cos(\omega t+\theta). $$

Now the relationship between $Y$ and the phase angle $\theta$ is non-linear, and you cannot regain linearity by the definition of a new feature.

Incidentally, I would encourage you to investigate Judea Pearl's concepts of causality in his book The Book of Why, and others. It will elucidate some of these ideas very clearly.