2
$\begingroup$

The formula for correlation coefficient is as follows:

$$\begin{align}\mathrm{corr} \left(\vec x, \vec y\right) = \frac{1}{n} \sum_{i=1}^n \frac{\left(x_i-\bar x\right)}{\sigma_x} \cdot \frac{\left(y_i - \bar y\right)}{\sigma_y} \end{align}$$

The idea behind correlation is the more similarly respective values in $\vec x$ and $\vec y$ deviates from their means, the closer $\mathbf{corr \left(\vec x, \vec y\right)}$ is to $\mathbf 1$. But how to read it off from this formula? It's not clear from the formula what it really calculates.


P.S. I don't know whether it's common on this site, but I've written the question to leave an answer. If it's not, let me know in comments.

$\endgroup$
6
  • 2
    $\begingroup$ I recommend your search around this site. This has been asked a ton of times and there are definitely answers available. $\endgroup$ Oct 30 '21 at 20:58
  • 1
    $\begingroup$ Please see stats.stackexchange.com/a/513608/919 for a deeper analysis of this issue. BTW, this is a strange formula for the correlation coefficient: usually the denominator is $n,$ not $n-1.$ With $n$ there, this clearly is the mean product of the standardized variables: that's straightforward to interpret. $\endgroup$
    – whuber
    Oct 30 '21 at 21:01
  • $\begingroup$ @Galen, Hello! I've already searched and haven't found anything similar. However, I can delete it, since I just wanted to share some intuition and don't actually need help. $\endgroup$
    – mathgeek
    Oct 30 '21 at 21:04
  • $\begingroup$ @whuber, Hello! I've written $n-1$, since standard deviation is usually calculated with $n-1$ denominator. Thank you for the link. Should I delete the post? $\endgroup$
    – mathgeek
    Oct 30 '21 at 21:06
  • 1
    $\begingroup$ You seem to be referring to an estimate of a standard deviation. The use of Greek variables $\sigma_x$ rather than latin variables $s_x$ implies you are referring to the SD itself rather than an estimate. $\endgroup$
    – whuber
    Oct 30 '21 at 21:08
4
$\begingroup$

Let's rewrite this formula as follows:

$$\begin{align} \mathrm{corr} \left(\vec x, \vec y\right) = \frac{\sum \left(x_i - \bar x\right)\left(y_i - \bar y\right)}{\sqrt{\sum\left(x_i - \bar x\right)^2}\sqrt{\sum\left(y_i - \bar y\right)^2}} = \\ = \frac{\left(\vec x - \bar x\right)}{\|\vec x - \bar x\|} \cdot \frac{{\left(\vec y - \bar y\right)}}{{ \|\vec y - \bar y\|}} = \\ = \cos \theta \end{align}$$

Hence $\mathrm{corr} \left(\vec x, \vec y\right)$ is just $\cos \theta$, where $\theta$ is the angle between vectors $\left(\vec x - \bar x\right)$ and $\left(\vec y - \bar y\right)$.

When we ask:

Why similar deviations of respective values in $\vec x$ and $\vec y$ from their means (expressed in standard units) make correlation coefficient closer to $1$?

by that we actually ask:

Why being vector $\frac{\left(\vec x - \bar x\right)}{\|\vec x - \bar x\|}$ closer to vector $\frac{{\left(\vec y - \bar y\right)}}{{ \|\vec y - \bar y\|}}$ makes the $\cos \theta$ closer to $1$?

since $\frac{\left(\vec x - \bar x\right)}{\|\vec x - \bar x\|}$ and $\frac{{\left(\vec y - \bar y\right)}}{{ \|\vec y - \bar y\|}}$ are those deviations (divided by $n$).


The takeaway is that when we calculate $\mathrm{corr} \left(\vec x, \vec y\right)$ we actually calculate the cosine of the angle between unit vectors $\frac{\left(\vec x - \bar x\right)}{\|\vec x - \bar x\|}$ and $\frac{{\left(\vec y - \bar y\right)}}{{ \|\vec y - \bar y\|}}$. Hence this angle tend to $0$ as respective deviations of vectors $\vec x$ and $\vec y$ from their means (i.e. coordinates of the unit vectors) tend to be the same.

$\endgroup$
7
  • 4
    $\begingroup$ You might enjoy reviewing the related posts that mention Anscombe's quartet. Although they refer to regression, they are relevant because the standardized least squares regression coefficient equals the correlation coefficient, thereby offering two more ways to interpret it (namely, by considering the regressions of $y$ vs $x$ and $x$ vs $y$). If you would like to explore this in more detail, my answer at stats.stackexchange.com/a/152034/919 provides software to produce more examples. $\endgroup$
    – whuber
    Oct 30 '21 at 21:22
  • $\begingroup$ sure, what doesn't look clearer on a n-dimensional space... $\endgroup$
    – carlo
    Oct 31 '21 at 12:30
  • 2
    $\begingroup$ @carlo Only two vectors are involved. According to Euclid, they generate a two dimensional space. (It's irrelevant that these vectors are written down using $n$ numbers each.) Thus, it's possible to draw a simple picture of the vectors using just a sheet of paper or a computer screen. $\endgroup$
    – whuber
    Oct 31 '21 at 15:58
  • 1
    $\begingroup$ @carlo, And here is this picture (for 3D). The sheet of paper is actually the span of two given vectors. $\endgroup$
    – mathgeek
    Oct 31 '21 at 16:48
  • 1
    $\begingroup$ @Carlo The formula lets us compute the angle. The Euclidean axioms let us see and conceptualize it. These advantages are available to any who would learn and practice the underlying concepts. $\endgroup$
    – whuber
    Oct 31 '21 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.