5
$\begingroup$

To quote Kevin Murphy in his book "Machine Learning - A probabilistic perspective", correlation is "a very limited measure of dependence". He talks about this before he introduces the concept of mutual information.

Why is the correlation coefficient "a limited measure of dependence"? Are there some assumptions associated with its computation?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Once you get a comfortable feeling for Pearson's correlation coefficient, and how it describes linear association, you might want to read Reshef, D., Reshef, Y., Finucane, H., Grossman, S., McVean, G., Turnbaugh, P., Lander, E., Mitzenmacher, M., and Sabeti, P. (2011). Detecting novel associations in large data sets. Science, 334(6062):1518–1524, as it is an approachable motivation of a really different approach (maximal information) to understanding how knowing something about X can tell you something about Y. $\endgroup$ – Alexis Mar 23 '18 at 16:39
11
$\begingroup$

This is explained in the Wikipedia entry for Correlation and Dependence. Correlation basically measures how close two variables are to having a linear relationship between them. Consider now $X \sim U(-1, 1)$, and $Y = X^2$. Then if you know $X$, you know $Y$ exactly, and if you know $Y$, you know $X$ up to its sign. Hence they are not independent. An easy calculation shows that their correlaton is 0, however.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ So basically what I understood is that correlation measures if or not there could be a linear relationship between X and Y. For instance if X depends on Y but not linearly, then correlation value between them is zero even though there is a great deal of dependence between the two. Am I right with this $\endgroup$ – Upendra Pratap Singh Mar 23 '18 at 11:26
  • $\begingroup$ In addition, how do you account for "if you know Y, you know X up to its sign"? $\endgroup$ – Upendra Pratap Singh Mar 23 '18 at 11:27
  • $\begingroup$ @UpendraPratapSingh You statement in the first comment is correct, IMHO. As for your second comment, suppose you know that $Y = {1 \over 2}$, then $X = \pm \sqrt{1 \over 2}$, for example. $\endgroup$ – Ami Tavory Mar 23 '18 at 12:33
  • 4
    $\begingroup$ If X depends non-linearly on Y, their correlation can be anything. Not necessarily 0. $\endgroup$ – Cris Luengo Mar 23 '18 at 13:03
  • 1
    $\begingroup$ @CrisLuengo That is correct, and thanks for the clarification. I think Upendra meant by "not linearly" completely not linearly, rather than not completely linearly. $\endgroup$ – Ami Tavory Mar 23 '18 at 13:11
1
$\begingroup$

A simple example. The correlation between a random variable $x$ and its square $x^2$ is zero for any symmetrical distribution on $\mathbb{R}$. Here's the means of a variable and its square: $$\mu=\int x dF(x)=0$$ $$\sigma^2=\int x^2 dF(x)$$

Let's calculate Pearson correlation: $$\rho=\frac{\int x x^2 dF(x)}{\mu \sigma^2}=\frac{\int x^3 dF(x)}{\mu \sigma^2}=\frac{0}{\mu \sigma^2}=0$$

However, if I know $x$ it tells me everything about $x^2$. That's one example where correlation does not reveal how much strong is the relationship between two variables.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ +1 There's probably a nuance to add about observed range of $x$, since even for standard normal, there's a non-zero relationship for ranges of $x$ dominated by either positive or negative values. $\endgroup$ – Alexis Mar 23 '18 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.