# Correlation in linear regression

I once heard the following statements about the "correlation" in the context of regression.

correlation only measures linear relationships

$\text{corr}(X,Y)=0$ does not mean the variables are not related!

My question is why can't correlation measure non-linear relationships?

Secondly, what does $\text{corr}(X,Y)=0$ tell us?

-
A good place to start would be by researching "Anscombe's quartet," which is shown in the answer at stats.stackexchange.com/a/16131. –  whuber Jul 14 at 20:35

As an opening comment, the word "correlation" sometimes is used in order to indicate the existence of an unspecified form of stochastic dependence -some authors will write "the variables are correlated" and they mean "the variables are dependent". So when one reads such a general statement, one must be careful to interpret it in the context that it is used.

To the other extreme, "correlation" sometimes is used as a shortcut for Pearson's product moment correlation coefficient which is the most widely used related measure, and for which the notation $\operatorname{Corr}(X,Y)$ is employed. But other measures do exist, so again, one should be alert.

The phrase "correlation only measures linear relationships" can be misleading, if one thinks hard enough about it. If it is zero, then we usually say "the relation between the two variables, if it exists is not linear". But if it is not zero, we cannot say that "their relation is linear". It can very well be non-linear and the correlation to be non-zero.

AN EXAMPLE

Consider a random variable $X$ that is continuous uniform $U(0,1)$ and define another random variable by $Y = X^3$. This is a clear non-linear relationship. Their covariance will be

$$\operatorname{Cov}(X,Y) = E(XY) - E(X)E(Y) = E(X^4) - E(X)E(X^3)$$

For a $U(0,1)$ random variable we have

$$E(X) = \frac 12,\;\; E(X^3) = \int_{0}^{1}x^3dx = \frac 14,\;\;E(X^4) = \int_{0}^{1}x^4dx = \frac 15$$

So $$\operatorname{Cov}(X,Y) = \frac 15 - \frac 12\cdot \frac 14 = \frac 3{40} \qquad [1]$$

The Pearson's correlation coefficient is

$$\operatorname{Corr}(X,Y) = \frac {\operatorname{Cov}(X,Y)}{\sqrt {\operatorname{Var}(X)\operatorname{Var}(Y)}}$$

We have

$$\operatorname{Var}(X) = \frac 1{12},\;\; \operatorname{Var}(Y)= E(X^6) - \left(E(X^3)\right)^2 = \frac 17 - \frac 1{16} = \frac 9{112}$$

Bringing it all together we get

$$\operatorname{Corr}(X,Y) = \frac {\frac 3{40}}{\sqrt {\frac 1{12}\cdot \frac 9{112}}} \approx 0.9165$$

... a very high value. Can we adequately express $Y$ as a linear function of $X$? Let's have a look: Based on a random sample of $1.000$ observations from a $U(0,1)$ we get

(note that the theoretical $\beta$ coefficient is $0.9$).

Is the blue straight line an "adequate representation" of the relation between $Y$ and $X$?

I would tentatively say that, if anything, the absolute value of the correlation coefficient measures the "uniformity" of the direction of covariance, rather than its "linear" nature. In our example, when $X$ tends to rise $Y$ rises too, in all cases (this is the nature of the non-linear relationship between them, given also the support of $X$).

-

Correlation is based on the covariance between two variables (it's basically scaled covariance). Covariance describes the linear relationship between two variables, as such correlation can only describe the linear relationship between two variables. This image should show you how a Cor(X,Y) = 0 can arise in multiple cases where there are no relationship and non-linear relationships between X and Y. Some of the examples are kind of silly, but the point about sinusoidal waves kind of drives it home.

Image by Denis Boigelot, given to the public domain. Source: Wikimedia Commons, https://commons.wikimedia.org/wiki/File:Correlation_examples2.svg

-
Nice. Note that you do not need continuous variables to describe a linear relationship. Indeed, it takes all of two different points to define as line on the Cartesian plane, if those two points happen to lie on integer values makes no never-mind. –  Alexis Jul 14 at 21:53
This image has been posted before on this site, so I'm pretty sure it is not original to you, Nate. Please include an attribution of its source in your answer. –  whuber Jul 15 at 0:28
@whuber, it's copied from Wikipedia. –  gung Jul 15 at 0:30
@gung I figured that, but the point is that nobody should be deceived about the origin of this image, nor should they have to hunt it down on the Web. We have the right of fair use to the work of others and one way we maintain that right is through full attribution. That principle applies even when work is in the public domain, as this one is. –  whuber Jul 15 at 0:35
Attributed. I have to say, I wonder if M. Boigelot riffed on Reshef's paper on the maximal information coefficient in Science. Some of those relationships look veeery familiar. –  Alexis Jul 15 at 1:05

$corr(X,Y)$ is just a scaled covariance so it evaluates, by construction, the linear co-dependence between the variables X and Y. To understand a little bit better how it works take a look at the covariance matrix formula. You'll realize it is a generalization of the variance formula, as it evaluates if two variables value vary in concordance or not. If they do, and their values increase together, the covariance will be positive, if when one decreases the onther decreases in the same way. If the way they vary is unrelated, then the average value calculated in the formula will be zero and we say the variables are not correlated.

You can say this analysis is simplistic, as it does not provide an evaluation of the correlation when different $Y(X)$ or $X(Y)$ are not assumed. Well, the problem is exactly that: to strictly evaluate another dependence you have to assume it, and if you prefer not to do so the analysis of correlation through the covariance is the most natural.

This said, you can evaluate the correlation using different indicators. Take a look at Spearman's Rank, in which you calculate the Pearson's correlation coefficient of the ranked variables, and and a consequence you do not assume a linear dependence but merely a monotonic one. Or even the more recent (and arguably more powerful) distance correlation. The wiki plots similar to the ones showed before will show you the different capabilities of these indicators.

-