Why does the correlation coefficient between X and X-Y random variables tend to be 0.7

Question

Taken from Practical Statistics for Medical Research where Douglas Altman writes in page 285:

...for any two quantities X and Y, X will be correlated with X-Y. Indeed, even if X and Y are samples of random numbers we would expect the correlation of X and X-Y to be 0.7

I tried this in R and it seems to be the case:

x <- rnorm(1000000, 10, 2)
y <- rnorm(1000000, 10, 2)
cor(x, x-y)

xu <- sample(1:100, size = 1000000, replace = T)
yu <- sample(1:100, size = 1000000, replace = T)
cor(xu, xu-yu)

Why is that? What is the theory behind this?

Answer 1

If $X$ and $Y$ are uncorrelated random variables with equal variance $\sigma^2$, then we have that $$\begin{align} \operatorname{var}(X-Y) &= \operatorname{var}(X) + \operatorname{var}(-Y)\\ &= \operatorname{var}(X) + \operatorname{var}(Y)\\ &=2\sigma^2,\\ \operatorname{cov}(X, X-Y) &= \operatorname{cov}(X,X) - \operatorname{cov}(X,Y) & \text{bilinearity of covariance operator}\\ &= \operatorname{var}(X) - 0 & 0 ~\text{because}~X ~\text{and}~ Y ~\text{are uncorrelated}\\ &= \sigma^2. \end{align}$$ Consequently, $$\rho_{X,X-Y} = \frac{\operatorname{cov}(X, X-Y)}{\sqrt{\operatorname{var}(X)\operatorname{var}(X-Y)}}= \frac{\sigma^2}{\sqrt{\sigma^2\cdot2\sigma^2}} = \frac{1}{\sqrt{2}}.$$ So, when you find $$\frac{\sum_{i=1}^n\left(x_i - \bar{x}\right) \left((x_i-y_i) - (\bar{x}-\bar{y})\right)}{ \sqrt{\sum_{i=1}^n\left(x_i - \bar{x}\right)^2 \sum_{i=1}^n\left((x_i-y_i) - (\bar{x}-\bar{y})\right)^2}} $$ the sample correlation of $x$ and $x-y$ for a large data set $\{(x_i,y_i)\colon 1 \leq i \leq n\}$ drawn from a population with these properties, which includes "random numbers" as a special case, the result tends to be close to the population correlation value $\frac{1}{\sqrt{2}} \approx 0.7071\ldots$

Answer 2

A geometrical-statistical explanation.

Imagine you make an "inside-out" scatterplot where the $n$ subjects are the axes and the $2$ variables $X$ and $Y$ are the points. This is called a subject space plot (as opposed to usual variable space plot). Because there is only 2 points to plot, all dimensions in such a space except just any two arbitrary dimensions that are able to support the 2 points plus the origin, are redundant and can be safely dropped. And so we are left with a plane. We draw vector arrows from the origin to the points: these are our variables $X$ and $Y$ as vectors in the subject space of the data.

Now, if the variables were centered then, in a subject space, the cosine of the angle between their vectors is their correlation coefficient. On the pic below $X$ and $Y$ vectors are orthogonal: their $r=0$. Uncorrelatedness was a prerequisite outlined by @Dilip in their answer.

Also for variables centered, their vector lengths in a subject space are their standard deviations. On the pic, $X$ and $Y$ are of equal length, - equal variances was also a prerequisite made by @Dilip.

To draw the variable $X-Y$ or variable $X+Y$ we just use vector addition or subtraction that we've forgotten since school (move Y vector over to the end of X vector and invert direction in case of subtraction, - this is shown by grey arrows on the pic, - then draw a vector to where the grey arrow points).

It becomes very clear that the length of $X-Y$ or $X+Y$ vectors (the standard deviation of these variables) is, by Pythagorean theorem, $\sqrt{2\sigma^2}$, and the angle between $X$ and $X-Y$ or $X+Y$ is 45 degrees, which cosine - the correlation - is $0.707...$

Answer 3

I believe that there's a simple intuition based on symmetry here, too. Since X and Y have the same distributions and have a covariance of 0, the relationship of X ± Y with X should "explain" half of the variation in X ± Y; the other half should be explained by Y. So R² should be 1/2, which means R is 1/√2 ≈ 0.707.

Answer 4

Here's a simple way to think about why there's a correlation here at all.

Imagine what goes on when you subtract two distributions. If the value of x is low then, on average, x - y will be a lower value than if the value of x is high. As x increases then x - y increase, on average, and thus, a positive correlation.