18
$\begingroup$

Let $X$, $Y$, and $Z$ are three random variables. Intuitively, I think that it is impossible to have $Cor(X, Y)=0.99$, $Cor(Y, Z)=0.99$ but $Cor(X, Z)=0$. My intuitive thought is that $X$ and $Z$ are nearly linearly correlated to $Y$. Hence, they are more or less linearly correlated, which makes the last equality impossible.

I pose this question because of the question and the comments (include my comments) here.

In general, as some others point out, I agree that it is possible that for some $\rho>0$ we may have $$Cor(X, Y)=\rho, Cor(Y, Z)=\rho \mbox{ and } Cor(X, Z)=0 \qquad (1).$$

My questions are:

  1. Do you think that (1) is wrong when $\rho$ is close to 1, e.g., 0.99?
  2. If (1) is wrong when $\rho$ is close to 1, what is the maximum value of $\rho$ so that (1) can be correct?
$\endgroup$
4
  • $\begingroup$ Couldn't this be achieved in the following scenario: Let $X$ be a uniform random binary variable, let $Z$ be a uniform random binary variable. Thus $Cor(X,Z) = 0$. Then let $Y = 2X + Z$. Then $Cor(X,Y) = Cor(Y,Z) = 1$ and if you added in some noise you could get that from 1 to 0.99 I think. This at least makes sense to me intuitively. $\endgroup$
    – ryan
    Nov 25, 2020 at 22:07
  • $\begingroup$ See the similar Q: stats.stackexchange.com/questions/131065/… $\endgroup$ Nov 26, 2020 at 1:13
  • 1
    $\begingroup$ @ryan - you may be confusing correlation and covariance. Your example does not lead to $Cor(X,Y) = Cor(Y,Z) = 1$ $\endgroup$
    – Henry
    Nov 30, 2020 at 10:11
  • $\begingroup$ @Henry, yes I was mistaken. My friend pointed out that the correlation would not be 1 since you cannot determine $Y$ from $X$ alone. Same thing for $Z$. $\endgroup$
    – ryan
    Dec 1, 2020 at 2:29

4 Answers 4

19
$\begingroup$

The correlation matrix needs to be positive semi-definite with non-negative eigenvalues. The eigenvalues of the correlation matrix are the solutions of $$ \left| \begin{matrix} 1-\lambda & \rho & \rho \\ \rho & 1-\lambda & 0 \\ \rho & 0 & 1-\lambda \end{matrix} \right| =(1-\lambda)\big((1-\lambda)^2-2\rho^2)\big) =0 $$ so the eigenvalues are $1$ and $1\pm\sqrt{2}\rho$. These are all non-negative for $$ -\frac1{\sqrt{2}} \le \rho \le \frac1{\sqrt{2}}. $$

$\endgroup$
8
$\begingroup$

A more intuitive perspective (an example) to complement @Jarle Tufto's +1 answer:

What you are asking, is whether something like this variance-covariance matrix is possible:

$\bf{\Sigma} = \begin{matrix} & X & Y & Z \\ X & 1 & 0.9 & 0 \\ Y & 0.9 & 1 & 0.9 \\ Z & 0 & 0.9 & 1\\ \end{matrix}$

This matrix is not positive-semidefinite. In fact, it is indefinite, since its determinant is negative. For example, a multivariate normal vector with this var-cov matrix cannot exist, since its PDF requires the determinant of $\Sigma$. If it is negative, the PDF would become negative, which would lead to negative probabilities. For this not to happen, the condition mentioned by @Jarle Tufto needs to be fulfilled.

$PDF_{Gauss}(x) =(2\pi)^{-0.5k}\det(\Sigma)^{-0.5}e^{-0.5(x-\mu)^T\Sigma^{-1}(x-\mu)}$

$\endgroup$
4
$\begingroup$

If you performed linear regression on $Y$, you would get an $R^2$ value of at most 1.

In your problem setting:

  • Performing linear regression on $Y$ with $X$ gets $R = 0.99$.
  • Performing linear regression on $Y$ with $Z$ gets $R = 0.99$.
  • $X$ and $Z$ are not correlated, so they would both independently contribute to the $R^2$ value of a regression on $Y$.

Combining these, when you perform linear regression on $Y$ with both $X$ and $Z$, you get $R^2 = (0.99)^2 + (0.99)^2 > 1$, which is impossible. This should also provide some idea of the bounds on these correlation values.

$\endgroup$
2
$\begingroup$

I posted this previously on Math StackExchange, but will reiterate here. If $\rho_{AB} = \text{Corr}(A, B)$, and similarly defined for $\rho_{BC}$ and $\rho_{AC}$, we have the inequality. \begin{align*} \rho_{AC} \ge \max\{2(\rho_{AB} + \rho_{BC}) - 3, 2\rho_{AB}\rho_{BC} - 1\} \end{align*} Proof. Some notation. I let $\sigma_{AB} = \text{Cov}(A,B)$ and $\sigma_A^2 = \text{Var}(A)$.

Let's first prove $\rho_{AC} \ge 2(\rho_{AB} + \rho_{BC}) - 3$. Recall the identity \begin{align*} 2 E[X^2] + 2E[Y^2] = E[(X+Y)^2] + E[(X-Y)^2] \end{align*} hence $2E[Y^2] \le E[(X+Y)^2] + E[(X-Y)^2]$. Set \begin{align*} X = \widetilde{B} - (\widetilde{A} + \widetilde{C})/2 \quad \text{and} \quad Y = (\widetilde{A} - \widetilde{C})/2 \end{align*} where $\widetilde{C} = (C - E[C])/\sigma_C$, the normalized random variable, and similarly for $\widetilde{A}, \widetilde{B}$. Upon substitution and simplification, we get \begin{align*} \frac{1}{2}(2 - 2\rho_{AC}) \le (2 - 2\rho_{AB}) + (2 - 2\rho_{BC}) \iff \rho_{AC} \ge 2(\rho_{AB} + \rho_{BC}) - 3 \end{align*} To prove $\rho_{AC} \ge 2\rho_{AB}\rho_{BC} - 1$, consider the random variable \begin{align*} W = 2 \frac{\sigma_{AB}}{\sigma_B^2}B - A \end{align*} We can verify $\sigma_W^2 = \sigma_A^2$, and hence $\sigma_{WC} \le \sigma_{W}\sigma_{C} = \sigma_ A \sigma_C$ by the Cauchy-Schwarz inequality. On the other hand, you may compute \begin{align*} \sigma_{WC} = 2 \frac{\sigma_{AB}}{\sigma_B^2}\sigma_{BC} - \sigma_{AC} \end{align*} Reorganizing all this, we prove $\rho_{AC} \ge 2\rho_{AB}\rho_{BC} - 1$.

In your specific example, with $\rho_{AB} = \rho_{BC} = 0.99$, then no matter the construction of $A, B, C$, we must have $\rho_{AC} \ge 0.9602$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.