Relationship between correlation and linear dependency

Question

I have two variables whose correlation coefficient $\rho=1$. I know that if two random variables are a linear combination of themselves, it implies $|\rho |=1$. Is the converse also true? How to prove that?

Answer 1

whuber's much more detailed answer appeared while I was composing this answer of mine (which essentially uses the same argument).

Let $X$ and $Y$ denote two random variables with finite variances $\sigma_X^2$ and $\sigma_Y^2$ respectively and correlation coefficient $\rho = \pm 1$. Then, \begin{align}\operatorname{var}(Y-aX) &= \sigma_Y^2+ a^2\sigma_X^2 - 2a\cdot\operatorname{cov}(Y,X) &\text{standard result}\\ &= \sigma_Y^2+ a^2\sigma_X^2 - 2a\rho\sigma_X\sigma_Y &\text{substitute for}~\operatorname{cov}(Y,X)\\ &= \sigma_Y^2+ a^2\sigma_X^2 \mp 2a\sigma_X\sigma_Y & \text{since}~ \rho = \pm 1\\ &= (\sigma_Y\mp a\sigma_X)^2\\ &= (\sigma_Y - a\rho\sigma_X)^2 & \text{keep remembering that}~ \rho = \pm 1\\ &= 0 &\text{if we choose}~ a = \rho\frac{\sigma_Y}{\sigma_X}. \end{align} Thus, if $\rho = \pm 1$, then $Y-\rho\frac{\sigma_Y}{\sigma_X}X$ is a random variable whose variance is $0$, and so $Y-\rho\frac{\sigma_Y}{\sigma_X}X$ is a constant (almost surely). In other words, $Y = \alpha X + \beta$ (almost surely) and thus $X$ and $Y$ are linearly related (almost surely).

Answer 2

Let the pair of (random) variables be $(X_1,X_2)$. Since their correlation coefficient exists, each has a finite variance $\sigma^2_i$ and a finite mean $\mu_i$. The standardized variables are $Z_i = (X_i - \mu_i)/\sigma_i$. In particular their second moments are unity:

$$\mathbb{E}(Z_i^2) = 1.$$

By definition, the correlation

$$\rho = \rho(X_1,X_2) = \mathbb{E}(Z_1Z_2)$$

is the expected product of the standardized variables. Since for any real numbers $x$, $y$, and $\rho$ it is the case that

$$2\rho x y = x^2 + \rho^2 y - (x-\rho y)^2,$$

and expectation is linear, we may compute

$$\eqalign{ 0 &= 2-2\rho^2 = 2 - 2\rho\, \mathbb{E}(Z_1Z_2) = 2- \mathbb{E}(2\rho Z_1Z_2) \\ &= 2-\mathbb{E}\left(Z_1^2 + Z_2^2- (Z_1 - \rho Z_2)^2\right) \\ &= 2-\left(1 + 1 - \mathbb{E}((Z_1 - \rho Z_2)^2)\right) \\ &= \mathbb{E}((Z_1 - \rho Z_2)^2). }$$

Now--for the first time--we need to invoke a basic result about random variables: because $(Z_1 - \rho Z_2)^2$ has zero expectation, $Z_1 - \rho Z_2 = 0$ almost surely. (A direct application of Chebyshev's Inequality will prove this.)

Unraveling the algebra to re-express the $Z_i$ in terms of the $X_i$, we find

$$\sigma_2 X_1 - \rho \sigma_1 X_2 + (\rho \sigma_1 \mu_2 - \sigma_2 \mu_1) = 0$$

almost surely. This is a linear relation between $X_1$ and $X_2$. It is not quite the result requested in the question, because it involves an additive constant $\rho \sigma_1 \mu_2 - \sigma_2 \mu_1$, whereas linear combinations do not. The constant is unavoidable: for instance, $X_1$ and $X_1+1$ have unit correlation but $X_1+1$ is not a linear combination of $X_1$.

What we have proven to be true is

When $|\rho(X_1,X_2)|=1$, $X_1$ and $X_2$ are linearly related almost surely. The coefficients in the relation are universal algebraic combinations of the first two moments of the variables.