If A and B are correlated with C, why are A and B not necessarily correlated?

I know empirically that is the case. I have just developed models that run into this conundrum. I also suspect it is not necessarily a yes/no answer. I mean by that if both A and B are correlated with C, this may have some implication regarding the correlation between A and B. But, this implication may be weak. It may be just a sign direction and nothing else.

Here is what I mean... Let's say A and B both have a 0.5 correlation with C. Given that, the correlation between A and B could well be 1.0. I think it also could be 0.5 or even lower. But, I think it is unlikely that it would be negative. Do you agree with that?

Also, is there an implication if you are considering the standard Pearson Correlation Coefficient or instead the Spearman (rank) Correlation Coefficient? My recent empirical observations were associated with the Spearman Correlation Coefficient.

-
An example is to take $A=X$, $B=Y$, and $C=X+Y$. We can take $X$ and $Y$ to be independent, yet both $A$ and $B$ are correlated (positively, Pearson) with $C$. –  G. Jay Kerns Dec 25 '10 at 23:51
Thanks, that's actually a great comment. Short, but it captures the essence of the reason why it is so. –  Gaetan Lion Dec 26 '10 at 0:07

Because correlation is a mathematical property of multivariate distributions, some insight can be had purely through calculations, regardless of the statistical genesis of those distributions.

For the Pearson correlations, consider multinormal variables $X$, $Y$, $Z$. These are useful to work with because any non-negative definite matrix actually is the covariance matrix of some multinormal distributions, thereby resolving the existence question. If we stick to matrices with $1$ on the diagonal, the off-diagonal entries of the covariance matrix will be their correlations. Writing the correlation of $X$ and $Y$ as $\rho$, the correlation of $Y$ and $Z$ as $\tau$, and the correlation of $X$ and $Z$ as $\sigma$, we compute that

• $1 + 2 \rho \sigma \tau - \left(\rho^2 + \sigma^2 + \tau^2\right) \ge 0$ (because this is the determinant of the correlation matrix and it cannot be negative).

• When $\sigma = 0$ this implies that $\rho^2 + \tau^2 \le 1$. To put it another way: when both $\rho$ and $\tau$ are large in magnitude, $X$ and $Z$ must have nonzero correlation.

• If $\rho^2 = \tau^2 = 1/2$, then any non-negative value of $\sigma$ (between $0$ and $1$ of course) is possible.

• When $\rho^2 + \tau^2 \lt 1$, negative values of $\sigma$ are allowable. For example, when $\rho = \tau = 1/2$, $\sigma$ can be anywhere between $-1/2$ and $1$.

These considerations imply there are indeed some constraints on the mutual correlations. The constraints (which depend only on the non-negative definiteness of the correlation matrix, not on the actual distributions of the variables) can be tightened depending on assumptions about the univariate distributions. For instance, it's easy to see (and to prove) that when the distributions of $X$ and $Y$ are not in the same location-scale family, their correlations must be strictly less than $1$ in size. (Proof: a correlation of $\pm 1$ implies $X$ and $Y$ are linearly related a.s.)

As far as Spearman rank correlations go, consider three trivariate observations $(1,1,2)$, $(2,3,1)$, and $(3,2,3)$ of $(X, Y, Z)$. Their mutual rank correlations are $1/2$, $1/2$, and $-1/2$. Thus even the sign of the rank correlation of $Y$ and $Z$ can be the reverse of the signs of the correlations of $X$ and $Y$ and $X$ and $Z$.

-
whuber, what are "multinormal variables"? –  Gaetan Lion Dec 26 '10 at 0:19
whuber, thanks the link was really helpful. –  Gaetan Lion Dec 28 '10 at 22:36
As usual, a most thorough explanation you get a well deserved "Best Answer" check mark. –  Gaetan Lion Dec 30 '10 at 21:17
@Gaetan Lion You are very kind. I have enjoyed reading all the answers to this question (and marked them all up). –  whuber Dec 30 '10 at 22:13

I'm on an annual fishing trip right now. There is a correlation between the time of day I fish and the amount of fish I catch. There is also a correlation between the size of the bait I use and the amount of fish I catch. There is no correlation between the size of the bait and the time of day.

-
Basil, I love it! +1 for a plain English explanation. –  Gaetan Lion Dec 28 '10 at 22:35
Best. Answer. On stats.stackexchange. Ever –  Chris Beeley Feb 9 '12 at 20:22

Correlation is the cosine of the angle between two vectors. In the situation described, (A,B,C) is a triple of observations, made n times, each observation being a real number. The correlation between A and B is the cosine of the angle between $V_A=A-E(A)$ and $V_B=B-E(B)$ as measured in n-dimensional euclidean space. So our situation reduces to considering 3 vectors $V_A$, $V_B$ and $V_C$ in n dimensional space. We have 3 pairs of vectors and therefore 3 angles. If two of the angles are small (high correlation) then the third one will also be small. But to say "correlated" is not much of a restriction: it means that the angle is between 0 and $\pi/2$. In general this gives no restriction at all on the third angle. Putting it another way, start with any angle less than $\pi$ between $V_A$ and $V_B$ (any correlation except -1). Let $V_C$ bisect the angle between $V_A$ and $V_B$. Then C will be correlated with both A and B.

-
+1 correlation in terms of an angle between multi-dimensional vectors is intuitive for me. –  pate Dec 26 '10 at 23:55
+1: Nice geometric explanation. –  whuber Dec 28 '10 at 22:52
For the reference of future readers, I expand on this geometric answer (with pictures!) in the following thread: talkstats.com/showthread.php/… –  Jake Westfall Sep 17 at 6:45

I will leave the statistical demonstration to those who are better suited than me for it... but intuitively say that event A generates a process X that contributes to the generation of event C. Then A is correlated to C (through X). B, on the other hand generates Y, that also shapes C. Therefore A is correlated to C, B is correlated to C but A and B are not correlated.

-
@Nice. I think you mean "A and B are not correlated" in the very last part of your last sentence. –  suncoolsu Dec 25 '10 at 21:05
Yes, Nico with suncoolsu correction... this is a reasonably good explanation. You are partially describing Path Analysis. –  Gaetan Lion Dec 25 '10 at 23:10
Yes, sorry, I got mixed up with the letters ;) –  nico Dec 26 '10 at 8:17

I think it's better to ask "why SHOULD they be correlated?" or, perhaps "Why should have any particular correlation?"

The following R code shows a case where x1 and x2 are both correlated with Y, but have 0 correlation with each other

x1 <- rnorm(100)
x2  <- rnorm(100)
y <- 3*x1 + 2*x2 + rnorm(100, 0, .3)

cor(x1,y)
cor(x2,y)
cor(x1,x2)


The correlation with Y can be made stronger by reducing the .3 to .1 or whatever

-
Unfortunately, I am not an R user. So, the codes above mean less to me than they mean to you. –  Gaetan Lion Dec 25 '10 at 23:07
@Gaetan Lion: in this code, $x_1$ and $x_2$ are independent root normals, and $y = 3x_1 + 2x_2$ plus a normal noise term with standard deviation of 0.3. Clearly $y$ is positively correlated to $x_1$ and $x_2$, which are independent. –  shabbychef Dec 26 '10 at 6:08

$1 + 2 \rho \sigma \tau - \left(\rho^2 + \sigma^2 + \tau^2\right) \ge 0$.

can be transformed into following inequality (Olkin, 1981):

$\sigma\tau - \sqrt{(1-\sigma^2)(1-\tau^2)} \le \rho \le \sigma\tau + \sqrt{(1-\sigma^2)(1-\tau^2)}$

A graphical representation of the upper and lower limits for $\rho$ looks like:

Olkin, I. (1981). Range restrictions for product-moment correlation matrices. Psychometrika, 46, 469-472. doi:10.1007/BF02293804

-