Why is the correlation coefficient a limited measure of dependence?

To quote Kevin Murphy in his book "Machine Learning - A probabilistic perspective", correlation is "a very limited measure of dependence". He talks about this before he introduces the concept of mutual information.

Why is the correlation coefficient "a limited measure of dependence"? Are there some assumptions associated with its computation?

• Once you get a comfortable feeling for Pearson's correlation coefficient, and how it describes linear association, you might want to read Reshef, D., Reshef, Y., Finucane, H., Grossman, S., McVean, G., Turnbaugh, P., Lander, E., Mitzenmacher, M., and Sabeti, P. (2011). Detecting novel associations in large data sets. Science, 334(6062):1518–1524, as it is an approachable motivation of a really different approach (maximal information) to understanding how knowing something about X can tell you something about Y. Commented Mar 23, 2018 at 16:39

This is explained in the Wikipedia entry for Correlation and Dependence. Correlation basically measures how close two variables are to having a linear relationship between them. Consider now $X \sim U(-1, 1)$, and $Y = X^2$. Then if you know $X$, you know $Y$ exactly, and if you know $Y$, you know $X$ up to its sign. Hence they are not independent. An easy calculation shows that their correlaton is 0, however.

• So basically what I understood is that correlation measures if or not there could be a linear relationship between X and Y. For instance if X depends on Y but not linearly, then correlation value between them is zero even though there is a great deal of dependence between the two. Am I right with this Commented Mar 23, 2018 at 11:26
• In addition, how do you account for "if you know Y, you know X up to its sign"? Commented Mar 23, 2018 at 11:27
• @UpendraPratapSingh You statement in the first comment is correct, IMHO. As for your second comment, suppose you know that $Y = {1 \over 2}$, then $X = \pm \sqrt{1 \over 2}$, for example. Commented Mar 23, 2018 at 12:33
• If X depends non-linearly on Y, their correlation can be anything. Not necessarily 0. Commented Mar 23, 2018 at 13:03
• @CrisLuengo That is correct, and thanks for the clarification. I think Upendra meant by "not linearly" completely not linearly, rather than not completely linearly. Commented Mar 23, 2018 at 13:11

A simple example. The correlation between a random variable $x$ and its square $x^2$ is zero for any symmetrical distribution on $\mathbb{R}$. Here's the means of a variable and its square: $$\mu=\int x dF(x)=0$$ $$\sigma^2=\int x^2 dF(x)$$

Let's calculate Pearson correlation: $$\rho=\frac{\int x x^2 dF(x)}{\mu \sigma^2}=\frac{\int x^3 dF(x)}{\mu \sigma^2}=\frac{0}{\mu \sigma^2}=0$$

However, if I know $x$ it tells me everything about $x^2$. That's one example where correlation does not reveal how much strong is the relationship between two variables.

• +1 There's probably a nuance to add about observed range of $x$, since even for standard normal, there's a non-zero relationship for ranges of $x$ dominated by either positive or negative values. Commented Mar 23, 2018 at 18:13