61
$\begingroup$

I know empirically that is the case. I have just developed models that run into this conundrum. I also suspect it is not necessarily a yes/no answer. I mean by that if both A and B are correlated with C, this may have some implication regarding the correlation between A and B. But, this implication may be weak. It may be just a sign direction and nothing else.

Here is what I mean... Let's say A and B both have a 0.5 correlation with C. Given that, the correlation between A and B could well be 1.0. I think it also could be 0.5 or even lower. But, I think it is unlikely that it would be negative. Do you agree with that?

Also, is there an implication if you are considering the standard Pearson Correlation Coefficient or instead the Spearman (rank) Correlation Coefficient? My recent empirical observations were associated with the Spearman Correlation Coefficient.

$\endgroup$
  • 37
    $\begingroup$ An example is to take $A=X$, $B=Y$, and $C=X+Y$. We can take $X$ and $Y$ to be independent, yet both $A$ and $B$ are correlated (positively, Pearson) with $C$. $\endgroup$ – user1108 Dec 25 '10 at 23:51
  • 1
    $\begingroup$ Thanks, that's actually a great comment. Short, but it captures the essence of the reason why it is so. $\endgroup$ – Sympa Dec 26 '10 at 0:07
51
$\begingroup$

Because correlation is a mathematical property of multivariate distributions, some insight can be had purely through calculations, regardless of the statistical genesis of those distributions.

For the Pearson correlations, consider multinormal variables $X$, $Y$, $Z$. These are useful to work with because any non-negative definite matrix actually is the covariance matrix of some multinormal distributions, thereby resolving the existence question. If we stick to matrices with $1$ on the diagonal, the off-diagonal entries of the covariance matrix will be their correlations. Writing the correlation of $X$ and $Y$ as $\rho$, the correlation of $Y$ and $Z$ as $\tau$, and the correlation of $X$ and $Z$ as $\sigma$, we compute that

  • $1 + 2 \rho \sigma \tau - \left(\rho^2 + \sigma^2 + \tau^2\right) \ge 0$ (because this is the determinant of the correlation matrix and it cannot be negative).

  • When $\sigma = 0$ this implies that $\rho^2 + \tau^2 \le 1$. To put it another way: when both $\rho$ and $\tau$ are large in magnitude, $X$ and $Z$ must have nonzero correlation.

  • If $\rho^2 = \tau^2 = 1/2$, then any non-negative value of $\sigma$ (between $0$ and $1$ of course) is possible.

  • When $\rho^2 + \tau^2 \lt 1$, negative values of $\sigma$ are allowable. For example, when $\rho = \tau = 1/2$, $\sigma$ can be anywhere between $-1/2$ and $1$.

These considerations imply there are indeed some constraints on the mutual correlations. The constraints (which depend only on the non-negative definiteness of the correlation matrix, not on the actual distributions of the variables) can be tightened depending on assumptions about the univariate distributions. For instance, it's easy to see (and to prove) that when the distributions of $X$ and $Y$ are not in the same location-scale family, their correlations must be strictly less than $1$ in size. (Proof: a correlation of $\pm 1$ implies $X$ and $Y$ are linearly related a.s.)

As far as Spearman rank correlations go, consider three trivariate observations $(1,1,2)$, $(2,3,1)$, and $(3,2,3)$ of $(X, Y, Z)$. Their mutual rank correlations are $1/2$, $1/2$, and $-1/2$. Thus even the sign of the rank correlation of $Y$ and $Z$ can be the reverse of the signs of the correlations of $X$ and $Y$ and $X$ and $Z$.

$\endgroup$
  • $\begingroup$ whuber, what are "multinormal variables"? $\endgroup$ – Sympa Dec 26 '10 at 0:19
  • 2
    $\begingroup$ en.wikipedia.org/wiki/Multivariate_normal_distribution $\endgroup$ – whuber Dec 26 '10 at 0:20
  • $\begingroup$ As usual, a most thorough explanation you get a well deserved "Best Answer" check mark. $\endgroup$ – Sympa Dec 30 '10 at 21:17
  • $\begingroup$ @Gaetan Lion You are very kind. I have enjoyed reading all the answers to this question (and marked them all up). $\endgroup$ – whuber Dec 30 '10 at 22:13
85
$\begingroup$

I'm on an annual fishing trip right now. There is a correlation between the time of day I fish and the amount of fish I catch. There is also a correlation between the size of the bait I use and the amount of fish I catch. There is no correlation between the size of the bait and the time of day.

$\endgroup$
  • $\begingroup$ Basil, I love it! +1 for a plain English explanation. $\endgroup$ – Sympa Dec 28 '10 at 22:35
  • $\begingroup$ Best. Answer. On stats.stackexchange. Ever $\endgroup$ – Chris Beeley Feb 9 '12 at 20:22
  • 1
    $\begingroup$ This describes a case where the correlations are low to begin with, but it doesn't explain the case where correlations are higher. If there's an 80% correlation with time of day, and there's an 80% correlation with the size of bait, I can guarantee that you're using bigger bait during the day! $\endgroup$ – user35581 Dec 8 '15 at 20:16
  • 2
    $\begingroup$ @user35581 no you can't--you're missing the whole point. Every hour he could fish once with small bait and once with large bait. He can still catch more fish during certain parts of the day (80% correlation) and catch more fish with larger bait (80% correlation) and there is 0 correlation between the size of bait he is using and the time of day. It could even be a negative correlation if he uses larger bait more often during off-peak times of day in order to compensate for the bad time of day. So you really know nothing about the correlation between time of day and size of bait. $\endgroup$ – rysqui Jan 5 '17 at 1:20
  • 1
    $\begingroup$ @rysqui sorry, my comment was poorly worded, but the point I was trying to make was this: when the correlations between features and target get very high, then your features must be correlated as well. So if you have a perfect correlation between time of day and size of catch, and a perfect correlation between size of bait and size of catch, then you must also have a perfect correlation between size of bait and time of day, hence the final statement "you're using bigger bait during the day". Keep in mind that this is an edge case! $\endgroup$ – user35581 Jan 6 '17 at 3:15
20
$\begingroup$

Correlation is the cosine of the angle between two vectors. In the situation described, (A,B,C) is a triple of observations, made n times, each observation being a real number. The correlation between A and B is the cosine of the angle between $V_A=A-E(A)$ and $V_B=B-E(B)$ as measured in n-dimensional euclidean space. So our situation reduces to considering 3 vectors $V_A$, $V_B$ and $V_C$ in n dimensional space. We have 3 pairs of vectors and therefore 3 angles. If two of the angles are small (high correlation) then the third one will also be small. But to say "correlated" is not much of a restriction: it means that the angle is between 0 and $\pi/2$. In general this gives no restriction at all on the third angle. Putting it another way, start with any angle less than $\pi$ between $V_A$ and $V_B$ (any correlation except -1). Let $V_C$ bisect the angle between $V_A$ and $V_B$. Then C will be correlated with both A and B.

$\endgroup$
  • $\begingroup$ +1 correlation in terms of an angle between multi-dimensional vectors is intuitive for me. $\endgroup$ – Petrus Theron Dec 26 '10 at 23:55
  • 2
    $\begingroup$ For the reference of future readers, I expand on this geometric answer (with pictures!) in the following thread: talkstats.com/showthread.php/… $\endgroup$ – Jake Westfall Sep 17 '13 at 6:45
18
$\begingroup$

As an add-on to whuber's answer: The presented formula

$1 + 2 \rho \sigma \tau - \left(\rho^2 + \sigma^2 + \tau^2\right) \ge 0$.

can be transformed into following inequality (Olkin, 1981):

$\sigma\tau - \sqrt{(1-\sigma^2)(1-\tau^2)} \le \rho \le \sigma\tau + \sqrt{(1-\sigma^2)(1-\tau^2)}$

A graphical representation of the upper and lower limits for $\rho$ looks like:

enter image description here


Olkin, I. (1981). Range restrictions for product-moment correlation matrices. Psychometrika, 46, 469-472. doi:10.1007/BF02293804

$\endgroup$
  • $\begingroup$ Can anyone tell me if some of these examples are multivariate distributions that have specific marginal distributions which restrict the range of possible correlations between components? That means the correlations cannot take the full range from -1 to 1. I recall that Frechet was at least one person that developed this in the 1950s. As I search the literature today I think they are now called Frechet copulas. $\endgroup$ – Michael Chernick Jan 2 '17 at 6:43
14
$\begingroup$

I think it's better to ask "why SHOULD they be correlated?" or, perhaps "Why should have any particular correlation?"

The following R code shows a case where x1 and x2 are both correlated with Y, but have 0 correlation with each other

x1 <- rnorm(100)
x2  <- rnorm(100)
y <- 3*x1 + 2*x2 + rnorm(100, 0, .3)

cor(x1,y)
cor(x2,y)
cor(x1,x2)

The correlation with Y can be made stronger by reducing the .3 to .1 or whatever

$\endgroup$
  • $\begingroup$ Unfortunately, I am not an R user. So, the codes above mean less to me than they mean to you. $\endgroup$ – Sympa Dec 25 '10 at 23:07
  • 2
    $\begingroup$ @Gaetan Lion: in this code, $x_1$ and $x_2$ are independent root normals, and $y = 3x_1 + 2x_2$ plus a normal noise term with standard deviation of 0.3. Clearly $y$ is positively correlated to $x_1$ and $x_2$, which are independent. $\endgroup$ – shabbychef Dec 26 '10 at 6:08
14
$\begingroup$

I will leave the statistical demonstration to those who are better suited than me for it... but intuitively say that event A generates a process X that contributes to the generation of event C. Then A is correlated to C (through X). B, on the other hand generates Y, that also shapes C. Therefore A is correlated to C, B is correlated to C but A and B are not correlated.

$\endgroup$
  • 1
    $\begingroup$ @Nice. I think you mean "A and B are not correlated" in the very last part of your last sentence. $\endgroup$ – suncoolsu Dec 25 '10 at 21:05
  • $\begingroup$ Yes, Nico with suncoolsu correction... this is a reasonably good explanation. You are partially describing Path Analysis. $\endgroup$ – Sympa Dec 25 '10 at 23:10
  • $\begingroup$ Yes, sorry, I got mixed up with the letters ;) $\endgroup$ – nico Dec 26 '10 at 8:17
1
$\begingroup$

For those who want some intuition, a correlation can be seen as a cosine of some angle. So, consider three vectors in 3D, let say A, B, and C, each corresponding to one variable. The question is to determine the range of possible angles between A and C when the angle between A and B as well as the angle between B et C are known. For that, you can play with an online tool without installing any software. Just go to the page http://www.montefiore.ulg.ac.be/~pierard/chained_correlations.php

$\endgroup$
0
$\begingroup$

Lets take one example:

A={x1,x2,x3,x4,x5,x6,x7,x8,x9}

B={x1,x2,x3,0,0,0,0,0,0}

C={0,0,0,x4,x5,x6,0,0,0}

For some x, A and B will have significant correlation, similarly A and C will also have significant correlation but correlation of B and C won't be significant.

So, It's not necessarily true that if A and B correlated and A and C are correlated then B and C are also correlated.

Note: For deep understanding, Please think this example on large data.

$\endgroup$
  • $\begingroup$ These assertions just aren't generally correct. $B$ and $C$ could be strongly correlated, depending on the values of $x1$ through $x6$. $A$ might or might not be correlated with $B$ or $C$, again depending on the values of $x1$ through $x9$. For "deep understanding," consider doing the calculations! $\endgroup$ – whuber Apr 4 '17 at 18:27
  • $\begingroup$ I am comfortable with Abhishek Anand answer because ultimately everything is correlated to everything else to some degree. And, I like the way he benchmarks it in terms of statistical significance. Once you use that framework it is pretty obvious that if A and B are statistically significantly correlated with C, either A or B may not necessarily be statistically significantly correlated (using the actual framework of my original question). I think vent diagrams can make for an excellent visual explanation of that concept. $\endgroup$ – Sympa Apr 5 '17 at 16:52
  • $\begingroup$ @whuber I agree with you. Its just one sample example which explain, why its not necessary $\endgroup$ – Abhishek Anand Apr 7 '17 at 13:10
  • $\begingroup$ That's fine--but you seem to have a misconception about what the correlations between these vectors are. None of the statements you make about the correlation coefficients of these vectors are generally correct. $\endgroup$ – whuber Apr 7 '17 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.