Is it possible to prove that Pearson coefficient is a measure of linear correlation between two variables?

I think I've seen proofs somewhere that if the relationship between variables $X, Y$ is given by linear equation $Y=aX+B$, then the absolute value of the Pearson coefficient equals $1$.

How to show that $0$ means they are not correlated at all, and intermediate values like $0.4$ suggest they are somehow more correlated, whatever that means? I think we must define first what is meant by stronger and weaker linear correlation between variables (and my question only makes sense if the Pearson coefficient itself is not this definition).

For instance, expected value is defined by a certain formula, and the law of large numbers theorem in a way 'proves' its intuitive meaning. I'm not aware of a proof that variance measures the spread of data, or Pearson coefficient measures the linear correlation.

  • 2
    You seem to be asking to prove something that is a definition. "Pearson coefficient" and "linear correlation" are two ends of the same stick, and I don't think you can prove one using the other. – Richard Hardy Nov 8 '15 at 19:11
  • 1
    Variance doesn't measure spread, but its square root (the standard deviation) clearly does, because (1) it's in the same units as the variable itself and (2) it's a weighted average of deviations from a center of the distribution (the mean). There are many other measures of spread, such as the $k^\text{th}$ root of the $k^\text{th}$ absolute moment around the mean (for any $k \ge 1$), the average of the $\alpha$ and $1-\alpha$ quantiles (for any $1/2\le\alpha\lt 1$), and many more. – whuber Nov 8 '15 at 23:22
  • @whuber You call standard deviation a 'weighted' average of deviations from the mean, because they are weighted by themselves (squared)? I.e. $\sqrt{\frac{\sum(X-\mu)^2}{N-1}}$. Regarding variance, why not just ignore the units? We can raise standard deviation to second, third, fourth power and it will be just equally good measure of spread if we don't care about units. – user4205580 Nov 9 '15 at 8:19
  • I think it's fair to say that without mathematical definition of what spread is, we can only treat every spread measure as its definition and pick one over the other depending on what we feel comfortable with in our model, for instance. – user4205580 Nov 9 '15 at 8:35
  • 1
    You lose a lot by ignoring the units, because then your measure of spread becomes arbitrary: it depends on the units in which the data are reported. I agree that each spread measure can be treated as a definition. I also agree that the choice of an appropriate definition should be appropriate for the kind of data, the underlying model, and the analytical objectives. – whuber Nov 9 '15 at 13:56

It is indeed possible to show that the Pearson correlation is essentially the way to measure linearity of association when you elect to use standard deviations to measure the dispersion of random variables.


Let's begin by noting that the Pearson correlation $\rho$ has to be considered as a (rather large) equivalence class of properties of bivariate random variables, because any invertible monotonic re-expression of it (such as its exponential) will carry identical information.

The question concerns what the term "linear relationship" might mean for random variables $(X,Y)$ that are not collinear. An important special case of this occurs when $(X,Y)$ is the empirical distribution of any bivariate dataset (which we may think of in terms of its scatterplot): are there natural ways to measure the departure of such a dataset (or scatterplot) from "linearity"?

Note that any invertible linear re-expression of the variables

$$(\xi,\eta) = (aX + b, cY + d)$$

(for constant $a,b,c,d$ where $a$ and $c$ are positive) will not change the linearity of their relationship (or lack thereof). We may therefore adopt some measure of "center" of a variable (such as its mean or median) and a measure of its "dispersion" around that center (such as its standard deviation or interquartile range) and stipulate that $a,b,c,d$ be chosen to place the centers of $\xi$ and $\eta$ at $0$ and scale their dispersions to unity.

If $(X,Y)$ are linearly related, this initial standardization will cause the support of $(\xi,\eta)$ to lie either on the line $\xi=\eta$ (for a positive relationship) or $\xi=-\eta$ (for a negative relationship). In the former case, the dispersion of $\xi-\eta$ is a natural measure of deviation from the line, while in the latter case the dispersion of $\xi+\eta$ measures deviation from the line. As a quantitative measurement of linearity, we may therefore compare one of these two quantities to the other. The larger this value is in size, the more linear is the original relationship between $X$ and $Y$.

Because two positive quantities are to be compared and a universal (unitless) measure is sought, the simplest way to make the comparison is the ratio. While the distributions of ratios tend to be skewed, the distributions of their logarithms tend not to be. Moreover, although a ratio cannot be negative, its logarithm potentially could be any real number.

Accordingly, linearity of relationship ought to be measured as the log ratio of dispersions of the sum and difference of the standardized variables.


As an example, consider using the first two moments to measure the center (the mean) and the dispersion (the standard deviation). The associated measure of linearity of $(X,Y)$ is

$$Z = \log \frac{\operatorname{SD}(\xi + \eta)}{\operatorname{SD} (\xi-\eta)} = \frac{1}{2}\log\frac{\operatorname{Var}(\xi+\eta)}{\operatorname{Var}(\xi-\eta)} = \frac{1}{2}\log \frac{2 + 2\rho}{2 - 2\rho} = \frac{1}{2}\log \frac{1 + \rho}{1 - \rho}$$

where $\rho$ is the Pearson correlation of $(X,Y)$. This expression for $Z$ is recognizable as the Fisher transformation of $\rho$, and therefore is equivalent to $\rho$ for assessing linearity. It is pleasing to see it drop out automatically from such basic principles.


This derivation has shown that the Pearson correlation coefficient is the natural way to measure the linearity of any bivariate distribution $(X,Y)$ when the first two moments are employed to evaluate central tendency and dispersion of variables.

One can go further and demonstrate that, among all the possible invertible monotonic transformations of $Z$, $\rho = \tanh(z)$ enjoys a special relationship to measures of linearity in simple ordinary least squares (OLS) regression: $\rho^2$ is identical to the coefficient of determination, $R^2$ in the regression of $Y$ against $X$ and in the regression of $X$ against $Y$. This is why $\rho$ rather than $Z$ is most often used, even in non-regression settings.

  • 2
    "It is pleasing to see it drop out automatically from such basic principles." Understatement. – Matthew Drury Nov 9 '15 at 0:17

Old question, I'm still searching for an answer though. Embrechts (1998) or Kruskal (1958) give this explanation, which may help, although to me it is not as clear as I would have hoped.

Let $ \alpha + \beta X $ be the best linear estimate of $ Y $ given $ X $ in the least square sense, i.e., $E[Y-a-bX]^2$ is minimized by $a=\alpha$ and $b=\beta$. Then $\beta = \text{Cov}(X,Y)/\text{Var}(X)$ and $\alpha = E(Y)-\beta E(X)$. It follows that $$ \rho^2(X,Y)= \frac{\text{Var}(Y)-E(Y-\alpha -\beta X)^2}{\text{Var}(Y)}. $$ This means that $\rho^2(X,Y)$ is the average relative reduction in the squared deviation of $ Y $ from its "best" linear estimate, relative to the marginal variance of $Y$.

  • Good contribution--welcome to our site! – whuber Mar 7 at 14:51

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.