49
$\begingroup$

Taken from Practical Statistics for Medical Research where Douglas Altman writes in page 285:

...for any two quantities X and Y, X will be correlated with X-Y. Indeed, even if X and Y are samples of random numbers we would expect the correlation of X and X-Y to be 0.7

I tried this in R and it seems to be the case:

x <- rnorm(1000000, 10, 2)
y <- rnorm(1000000, 10, 2)
cor(x, x-y)

xu <- sample(1:100, size = 1000000, replace = T)
yu <- sample(1:100, size = 1000000, replace = T)
cor(xu, xu-yu)

Why is that? What is the theory behind this?

$\endgroup$
  • $\begingroup$ What part do you want an explanation for? Do you just want the simplified equation for the correlation that results because of the known correlation between x, and y, and covariance between x and x-y? Or, do you just want to know why there's any covariance here at all? $\endgroup$ – John Mar 6 '13 at 11:52
  • $\begingroup$ Is this true for any $X$ and $Y$? Suppose $X$ and $Z$ are uncorrelated and let $Y=X-Z$. Then I suspect $X$ will not be correlated with $X-Y$. $\endgroup$ – Henry Aug 23 '13 at 17:08
69
$\begingroup$

If $X$ and $Y$ are uncorrelated random variables with equal variance $\sigma^2$, then we have that $$\begin{align} \operatorname{var}(X-Y) &= \operatorname{var}(X) + \operatorname{var}(-Y)\\ &= \operatorname{var}(X) + \operatorname{var}(Y)\\ &=2\sigma^2,\\ \operatorname{cov}(X, X-Y) &= \operatorname{cov}(X,X) - \operatorname{cov}(X,Y) & \text{bilinearity of covariance operator}\\ &= \operatorname{var}(X) - 0 & 0 ~\text{because}~X ~\text{and}~ Y ~\text{are uncorrelated}\\ &= \sigma^2. \end{align}$$ Consequently, $$\rho_{X,X-Y} = \frac{\operatorname{cov}(X, X-Y)}{\sqrt{\operatorname{var}(X)\operatorname{var}(X-Y)}}= \frac{\sigma^2}{\sqrt{\sigma^2\cdot2\sigma^2}} = \frac{1}{\sqrt{2}}.$$ So, when you find $$\frac{\sum_{i=1}^n\left(x_i - \bar{x}\right) \left((x_i-y_i) - (\bar{x}-\bar{y})\right)}{ \sqrt{\sum_{i=1}^n\left(x_i - \bar{x}\right)^2 \sum_{i=1}^n\left((x_i-y_i) - (\bar{x}-\bar{y})\right)^2}} $$ the sample correlation of $x$ and $x-y$ for a large data set $\{(x_i,y_i)\colon 1 \leq i \leq n\}$ drawn from a population with these properties, which includes "random numbers" as a special case, the result tends to be close to the population correlation value $\frac{1}{\sqrt{2}} \approx 0.7071\ldots$

$\endgroup$
  • $\begingroup$ Could you please explain a bit more how cov(X,X)-cov(X,Y)=s^2 $\endgroup$ – nostock Mar 6 '13 at 18:56
  • 5
    $\begingroup$ cov(X,X) is another name for var(X). cov(X,Y) = 0 since X and Y are assumed to be uncorrelated (hence covariance = 0). $\endgroup$ – Dilip Sarwate Mar 6 '13 at 19:15
58
$\begingroup$

A geometrical-statistical explanation.

Imagine you make an "inside-out" scatterplot where the $n$ subjects are the axes and the $2$ variables $X$ and $Y$ are the points. This is called a subject space plot (as opposed to usual variable space plot). Because there is only 2 points to plot, all dimensions in such a space except just any two arbitrary dimensions that are able to support the 2 points plus the origin, are redundant and can be safely dropped. And so we are left with a plane. We draw vector arrows from the origin to the points: these are our variables $X$ and $Y$ as vectors in the subject space of the data.

Now, if the variables were centered then, in a subject space, the cosine of the angle between their vectors is their correlation coefficient. On the pic below $X$ and $Y$ vectors are orthogonal: their $r=0$. Uncorrelatedness was a prerequisite outlined by @Dilip in their answer.

Also for variables centered, their vector lengths in a subject space are their standard deviations. On the pic, $X$ and $Y$ are of equal length, - equal variances was also a prerequisite made by @Dilip.

To draw the variable $X-Y$ or variable $X+Y$ we just use vector addition or subtraction that we've forgotten since school (move Y vector over to the end of X vector and invert direction in case of subtraction, - this is shown by grey arrows on the pic, - then draw a vector to where the grey arrow points).

It becomes very clear that the length of $X-Y$ or $X+Y$ vectors (the standard deviation of these variables) is, by Pythagorean theorem, $\sqrt{2\sigma^2}$, and the angle between $X$ and $X-Y$ or $X+Y$ is 45 degrees, which cosine - the correlation - is $0.707...$

enter image description here

$\endgroup$
  • 4
    $\begingroup$ A big +1 for sharing this approach. $\endgroup$ – whuber Mar 6 '13 at 16:43
  • $\begingroup$ (+1) That is a very neat way of presenting this! $\endgroup$ – Matt Krause Mar 6 '13 at 17:11
  • $\begingroup$ Ahh...pictures! (+1) Well done. :-) $\endgroup$ – cardinal Mar 11 '13 at 13:45
11
$\begingroup$

I believe that there's a simple intuition based on symmetry here, too. Since X and Y have the same distributions and have a covariance of 0, the relationship of X ± Y with X should "explain" half of the variation in X ± Y; the other half should be explained by Y. So R2 should be 1/2, which means R is 1/√2 ≈ 0.707.

$\endgroup$
  • $\begingroup$ This seems like a nice intuition, but note that if $r^2=\frac 1 2$, the standard way to write $r$ would be $\sqrt{1/2}$, not $1/\sqrt 2$ which might confuse some people even if they are algebraically equivalent. $\endgroup$ – gung Mar 30 '14 at 8:05
  • $\begingroup$ No, that really is not more standard. (If you need evidence, look up at the top answer. The 38 people who have already voted for it didn't quibble with the same notation.) $\endgroup$ – denn333 Mar 30 '14 at 16:19
  • $\begingroup$ I'm one of those 38 ;-). The question is, what will somebody whose algebra is fairly weak be most easily able to follow? If $r^2=1/2$, then it is easier to see that $r=\sqrt{1/2}$. $\endgroup$ – gung Mar 30 '14 at 16:24
3
$\begingroup$

Here's a simple way to think about why there's a correlation here at all.

Imagine what goes on when you subtract two distributions. If the value of x is low then, on average, x - y will be a lower value than if the value of x is high. As x increases then x - y increase, on average, and thus, a positive correlation.

$\endgroup$
  • 4
    $\begingroup$ I don't think your statement is always true "There will always be a correlation between two random distributions when there's a mathematical relationship." e.g. x <- rnorm(1e6, 0,1) y <- rnorm(1e6, 0,1) $cor((x-y)^2,x-y) $ $\endgroup$ – curious_cat Mar 6 '13 at 13:34
  • 4
    $\begingroup$ @curious_cat: Or, perhaps to be even more evocative, drop the y altogether. :-) $\endgroup$ – cardinal Mar 11 '13 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.