Relationship between correlation and linear dependency I have two variables whose correlation coefficient $\rho=1$. I know that if two random variables are a linear combination of themselves, it implies $|\rho |=1$. Is the converse also true?  How to prove that? 
 A: Let the pair of (random) variables be $(X_1,X_2)$.  Since their correlation coefficient exists, each has a finite variance $\sigma^2_i$ and a finite mean $\mu_i$.  The standardized variables are $Z_i = (X_i - \mu_i)/\sigma_i$.  In particular their second moments are unity:
$$\mathbb{E}(Z_i^2) = 1.$$
By definition, the correlation
$$\rho = \rho(X_1,X_2) = \mathbb{E}(Z_1Z_2)$$
is the expected product of the standardized variables.  Since for any real numbers $x$, $y$, and $\rho$ it is the case that 
$$2\rho x y =  x^2 + \rho^2 y - (x-\rho y)^2,$$
and expectation is linear, we may compute
$$\eqalign{
0 &= 2-2\rho^2 = 2 - 2\rho\, \mathbb{E}(Z_1Z_2) = 2- \mathbb{E}(2\rho Z_1Z_2) \\
&= 2-\mathbb{E}\left(Z_1^2 + Z_2^2- (Z_1 - \rho Z_2)^2\right) \\
&= 2-\left(1 + 1 - \mathbb{E}((Z_1 - \rho Z_2)^2)\right) \\
&= \mathbb{E}((Z_1 - \rho Z_2)^2).
}$$
Now--for the first time--we need to invoke a basic result about random variables: because $(Z_1 - \rho Z_2)^2$ has zero expectation, $Z_1 - \rho Z_2 = 0$ almost surely. (A direct application of Chebyshev's Inequality will prove this.)
Unraveling the algebra to re-express the $Z_i$ in terms of the $X_i$, we find
$$\sigma_2 X_1 - \rho \sigma_1 X_2 + (\rho \sigma_1 \mu_2 - \sigma_2 \mu_1) = 0$$
almost surely.  This is a linear relation between $X_1$ and $X_2$.  It is not quite the result requested in the question, because it involves an additive constant $\rho \sigma_1 \mu_2 - \sigma_2 \mu_1$, whereas linear combinations do not. The constant is unavoidable: for instance, $X_1$ and $X_1+1$ have unit correlation but $X_1+1$ is not a linear combination of $X_1$.
What we have proven to be true is

When $|\rho(X_1,X_2)|=1$, $X_1$ and $X_2$ are linearly related almost surely.  The coefficients in the relation are universal algebraic combinations of the first two moments of the variables.

A: whuber's much more detailed answer appeared while I was composing this answer of mine (which essentially uses the same argument).
Let $X$ and $Y$ denote two random variables with finite variances
$\sigma_X^2$ and $\sigma_Y^2$ respectively and correlation coefficient
$\rho = \pm 1$.  Then, 
\begin{align}\operatorname{var}(Y-aX) 
&= \sigma_Y^2+ a^2\sigma_X^2 - 2a\cdot\operatorname{cov}(Y,X)
&\text{standard result}\\
&= \sigma_Y^2+ a^2\sigma_X^2 - 2a\rho\sigma_X\sigma_Y
&\text{substitute for}~\operatorname{cov}(Y,X)\\
&= \sigma_Y^2+ a^2\sigma_X^2 \mp 2a\sigma_X\sigma_Y
& \text{since}~ \rho = \pm 1\\
&= (\sigma_Y\mp a\sigma_X)^2\\
&= (\sigma_Y - a\rho\sigma_X)^2
& \text{keep remembering that}~ \rho = \pm 1\\
&= 0 &\text{if we choose}~ a = \rho\frac{\sigma_Y}{\sigma_X}.
\end{align}
Thus, if $\rho = \pm 1$, then $Y-\rho\frac{\sigma_Y}{\sigma_X}X$
is a random
variable whose variance is $0$, and so
$Y-\rho\frac{\sigma_Y}{\sigma_X}X$ is a constant
(almost surely). In other words, $Y = \alpha X + \beta$ (almost
surely) and thus $X$ and $Y$ are linearly related
(almost surely).
