11
$\begingroup$

Is it possible to prove that Pearson coefficient is a measure of linear correlation between two variables?

I think I've seen proofs somewhere that if the relationship between variables $X, Y$ is given by linear equation $Y=aX+B$, then the absolute value of the Pearson coefficient equals $1$.

How to show that $0$ means they are not correlated at all, and intermediate values like $0.4$ suggest they are somehow more correlated, whatever that means? I think we must define first what is meant by stronger and weaker linear correlation between variables (and my question only makes sense if the Pearson coefficient itself is not this definition).

For instance, expected value is defined by a certain formula, and the law of large numbers theorem in a way 'proves' its intuitive meaning. I'm not aware of a proof that variance measures the spread of data, or Pearson coefficient measures the linear correlation.

$\endgroup$
7
  • 2
    $\begingroup$ You seem to be asking to prove something that is a definition. "Pearson coefficient" and "linear correlation" are two ends of the same stick, and I don't think you can prove one using the other. $\endgroup$ Nov 8, 2015 at 19:11
  • 2
    $\begingroup$ Variance doesn't measure spread, but its square root (the standard deviation) clearly does, because (1) it's in the same units as the variable itself and (2) it's a weighted average of deviations from a center of the distribution (the mean). There are many other measures of spread, such as the $k^\text{th}$ root of the $k^\text{th}$ absolute moment around the mean (for any $k \ge 1$), the average of the $\alpha$ and $1-\alpha$ quantiles (for any $1/2\le\alpha\lt 1$), and many more. $\endgroup$
    – whuber
    Nov 8, 2015 at 23:22
  • $\begingroup$ @whuber You call standard deviation a 'weighted' average of deviations from the mean, because they are weighted by themselves (squared)? I.e. $\sqrt{\frac{\sum(X-\mu)^2}{N-1}}$. Regarding variance, why not just ignore the units? We can raise standard deviation to second, third, fourth power and it will be just equally good measure of spread if we don't care about units. $\endgroup$ Nov 9, 2015 at 8:19
  • $\begingroup$ I think it's fair to say that without mathematical definition of what spread is, we can only treat every spread measure as its definition and pick one over the other depending on what we feel comfortable with in our model, for instance. $\endgroup$ Nov 9, 2015 at 8:35
  • 1
    $\begingroup$ You lose a lot by ignoring the units, because then your measure of spread becomes arbitrary: it depends on the units in which the data are reported. I agree that each spread measure can be treated as a definition. I also agree that the choice of an appropriate definition should be appropriate for the kind of data, the underlying model, and the analytical objectives. $\endgroup$
    – whuber
    Nov 9, 2015 at 13:56

3 Answers 3

11
$\begingroup$

It is indeed possible to show that the Pearson correlation is essentially the way to measure linearity of association when you elect to use standard deviations to measure the dispersion of random variables.


Let's begin by noting that the Pearson correlation $\rho$ has to be considered as a (rather large) equivalence class of properties of bivariate random variables, because any invertible monotonic re-expression of it (such as its exponential) will carry identical information.

The question concerns what the term "linear relationship" might mean for random variables $(X,Y)$ that are not collinear. An important special case of this occurs when $(X,Y)$ is the empirical distribution of any bivariate dataset (which we may think of in terms of its scatterplot): are there natural ways to measure the departure of such a dataset (or scatterplot) from "linearity"?

Note that any invertible linear re-expression of the variables

$$(\xi,\eta) = (aX + b, cY + d)$$

(for constant $a,b,c,d$ where $a$ and $c$ are positive) will not change the linearity of their relationship (or lack thereof). We may therefore adopt some measure of "center" of a variable (such as its mean or median) and a measure of its "dispersion" around that center (such as its standard deviation or interquartile range) and stipulate that $a,b,c,d$ be chosen to place the centers of $\xi$ and $\eta$ at $0$ and scale their dispersions to unity.

If $(X,Y)$ are linearly related, this initial standardization will cause the support of $(\xi,\eta)$ to lie either on the line $\xi=\eta$ (for a positive relationship) or $\xi=-\eta$ (for a negative relationship). In the former case, the dispersion of $\xi-\eta$ is a natural measure of deviation from the line, while in the latter case the dispersion of $\xi+\eta$ measures deviation from the line. As a quantitative measurement of linearity, we may therefore compare one of these two quantities to the other. The larger this value is in size, the more linear is the original relationship between $X$ and $Y$.

Because two positive quantities are to be compared and a universal (unitless) measure is sought, the simplest way to make the comparison is the ratio. While the distributions of ratios tend to be skewed, the distributions of their logarithms tend not to be. Moreover, although a ratio cannot be negative, its logarithm potentially could be any real number.

Accordingly, linearity of relationship ought to be measured as the log ratio of dispersions of the sum and difference of the standardized variables.


As an example, consider using the first two moments to measure the center (the mean) and the dispersion (the standard deviation). The associated measure of linearity of $(X,Y)$ is

$$Z = \log \frac{\operatorname{SD}(\xi + \eta)}{\operatorname{SD} (\xi-\eta)} = \frac{1}{2}\log\frac{\operatorname{Var}(\xi+\eta)}{\operatorname{Var}(\xi-\eta)} = \frac{1}{2}\log \frac{2 + 2\rho}{2 - 2\rho} = \frac{1}{2}\log \frac{1 + \rho}{1 - \rho}$$

where $\rho$ is the Pearson correlation of $(X,Y)$. This expression for $Z$ is recognizable as the Fisher transformation of $\rho$, and therefore is equivalent to $\rho$ for assessing linearity. It is pleasing to see it drop out automatically from such basic principles.


This derivation has shown that the Pearson correlation coefficient is the natural way to measure the linearity of any bivariate distribution $(X,Y)$ when the first two moments are employed to evaluate central tendency and dispersion of variables.

One can go further and demonstrate that, among all the possible invertible monotonic transformations of $Z$, $\rho = \tanh(z)$ enjoys a special relationship to measures of linearity in simple ordinary least squares (OLS) regression: $\rho^2$ is identical to the coefficient of determination, $R^2$ in the regression of $Y$ against $X$ and in the regression of $X$ against $Y$. This is why $\rho$ rather than $Z$ is most often used, even in non-regression settings.

$\endgroup$
4
  • 4
    $\begingroup$ "It is pleasing to see it drop out automatically from such basic principles." Understatement. $\endgroup$ Nov 9, 2015 at 0:17
  • $\begingroup$ do you have a reference for "distribution of ratios tends to be skewed"? (or an intuition of what and why that is) Btw, nice answer. $\endgroup$ Oct 13, 2021 at 18:53
  • $\begingroup$ btw, out of curiosity, how do we measure/quantify non-linear relationships (in one metric)? I wonder what the ratio of polynomials of Pearsons-correlations means... $\endgroup$ Oct 13, 2021 at 19:06
  • 1
    $\begingroup$ @Charlie "Nonlinearity" is so general it would be hopeless to quantify it with a single scalar (if that's what you might mean by "one metric"). I recently ran across one interesting attempt to do it using a plot: jstor.org/stable/1403769. I also submitted another proposal to a stats journal last year, based on the visualizations at stats.stackexchange.com/a/18200/919, but the editor rejected it as being too obvious. $\endgroup$
    – whuber
    Oct 13, 2021 at 21:05
6
$\begingroup$

Old question, I'm still searching for an answer though. Embrechts (1998) or Kruskal (1958) give this explanation, which may help, although to me it is not as clear as I would have hoped.

Let $ \alpha + \beta X $ be the best linear estimate of $ Y $ given $ X $ in the least square sense, i.e., $E[Y-a-bX]^2$ is minimized by $a=\alpha$ and $b=\beta$. Then $\beta = \text{Cov}(X,Y)/\text{Var}(X)$ and $\alpha = E(Y)-\beta E(X)$. It follows that $$ \rho^2(X,Y)= \frac{\text{Var}(Y)-E(Y-\alpha -\beta X)^2}{\text{Var}(Y)}. $$ This means that $\rho^2(X,Y)$ is the average relative reduction in the squared deviation of $ Y $ from its "best" linear estimate, relative to the marginal variance of $Y$.

$\endgroup$
3
  • $\begingroup$ Good contribution--welcome to our site! $\endgroup$
    – whuber
    Mar 7, 2018 at 14:51
  • $\begingroup$ @A-Sus: This is insightful. Can you please help me understand how you get the expression for the square of correlation? $\endgroup$ Dec 10, 2018 at 13:27
  • 1
    $\begingroup$ @ColorStatistics: sorry for the late reply. By expressing $\rho^2$ using the covariance and substituting $Cov(X,Y)^2=\beta^2 Var(X)^2$ you should easily get to $\rho^2=Var(\beta X) / Var(Y)$. Then you only need to prove that the numerator of the fraction equals $Var(\beta X)$. That's because $E(Y-\alpha-\beta X)^2 = Var(Y-\alpha-\beta X) + [E(Y-\alpha-\beta X) ]^2$ and the last expectation is zero (see the expression of $\alpha$). Then use the formula for the variance of the sum of $Y$ and $\alpha+\beta X$ and notice that $Cov(Y, \beta X) = \beta^2 Var(X) = Var (\beta X) $. $\endgroup$
    – A-Sus
    Dec 30, 2018 at 13:13
1
$\begingroup$

geometric interpretation

I found this very helpful, hope it helps others! more details visit the link. So if the Pearson coefficient = 0, the two line will be perpendicular.

https://en.wikipedia.org/wiki/Pearson_correlation_coefficient#Interpretation

$\endgroup$
2
  • $\begingroup$ Does that text prove what was asked, namely that Pearson is a measure of linear correlation? $\endgroup$
    – mdewey
    Oct 18, 2020 at 14:22
  • $\begingroup$ @mdewey I would say It's more like a geometric intuition 'proof', it explains why 0 means no correlation at all(as the question asked), since geometrically 0 correlation means two vectors are perpendicular. look at the formula of vector dot product and Pearson correlation, they are basically the same. $\endgroup$ Oct 20, 2020 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.