Is "pearson correlation higher than x" transitive?

Question

Consider a set of features $x_1,x_2,...x_N$ that may be (pearson-) correlated.

I want to group them whenever they have a high correlation between themselves so that I can interpret the coefficients of a linear regression on them.

To this, I constructed an undirected (because correlation is commutative) graph/network where feature $x_i$ is connected to feature $x_j$ when they have a correlation above some threshold.

The task of grouping correlated features amounts to find the components of this graph.

Can I assume that if the edges $(x_1,x_2)$ and $(x_1,x_3)$ exist, then $(x_2,x_3)$ exists too?

I.e. Does $corr(x_1,x_2) \geq z$ and $corr(x_1,x_3) \geq z$ imply that $corr(x_2,x_3) \geq z$?

Answer 1

No, correlation is not transitive.

Consider:

X1 = [0,1,2,0]

X2 = [0,1,1,0]

X3 = [0,2,1,0]

Then:

corr(X1,X2) = 0.90

corr(X2,X3) = 0.90

corr(X1,X3) = 0.64

Therefore choosing a cutoff of x=0.8 will add an edge from X1 to X2 and from X2 to X3, but not from X1 to X3.

To make your feature-grouping approach consistent, you'll need to first add the edges above the correlation cutoff, and then identify the maximal connected components in the graph.

I followed an identical process for merging correlated features in a recent analysis, and it worked well.

Answer 2

No, you cannot assume that. Consider three random variables $x$, $y$, and $z$ such that:

\begin{align} x=\rho\ z + \sqrt{1-\rho^2}\ \epsilon_1 \\ y=\rho\ z + \sqrt{1-\rho^2}\ \epsilon_2 \end{align}

where $Corr(z,\epsilon_1)=Corr(z,\epsilon_2)=0$.

Then $Corr(x,z)=Corr(y,z)=\rho$, but $Corr(x,y)=\rho^2+(1-\rho^2)Corr(\epsilon_1,\epsilon_2)$, which implies $2\rho^2-1\le Corr(x,y) \le 1$.

Answer 3

Let A = $corr(x_1,x_2)$, B = $corr(x_1,x_3)$, C = $corr(x_2,x_3)$. Your question is:

"Does A > z and B > z imply C > z?"

We don't actually need much knowledge of statistics to answer that question. Given any three distinct numbers, regardless of how they are generated, there must be some number z such that two of the numbers are greater than z, but one of them is not.

So there are only two questions we need to answer. First, are A,B,C distinct? Theoretically, they could be equal, but since we are asking whether communitivity always holds, it suffices to consider cases where the numbers are distinct, and thus for the purposes of showing a counterexample we can assume that they are distinct. The second issue is whether there is anything special about the order A,B,C; the statement "there is some number z such that two of the numbers are greater than z, but the third is not" depends on us being able to choose which is the "third" number, so if there's something special about C, then this could make our counterexample invalid. But there is not in fact anything special about any of the numbers; the indices on $x_1$, $x_2$, and $x_3$ are arbitrary, and therefore which correlation is A,B, or C is also arbitrary.

Answer 4

Say $X=(x_1,x_2,x_3)$ and $Y=(y_1,y_2,y_3)$. Suppose also that the means of $X$ and $Y$ have already been subtracted so that $\sum x_i = \sum y_i=0$. Then the standard deviation of $X$ is essentially the length of $X$ as a vector, similarly for $Y$. Meanwhile the covariance is the dot product, which can be expressed as $|X||Y|\cos \theta$, where $\theta$ is the angle between the vectors. Thus the correlation is just the cosine of the angle between the vectors.

Translating your question into the language of vectors, you are essentially asking if the angle between $X$ and $Y$ is less than ten degrees, and the angle between $Y$ and $Z$ is less than ten degrees, does the angle between $X$ and $Z$ have to be less than ten degrees? The answer of course is no. Note the logic works just as well in higher dimensions as it does in three.

Answer 5

Pearson correlation is NOT transitive.

For example, let $corr(x_1,x_2) = corr(x_1,x_3) = 0.5$ . Then $corr(x_2,x_3)$ can be any value in the range [-0.5,1].

This can be seen by arranging the known correlations into a 3 by 3 matrix, with $corr(x_2,x_3)$ as an unknown (variable) in the matrix. The unknown (occupying 2 locations in the matrix) can be any value such that the correlation matrix is positive semidefinite. Using semidefinite programming (Linear Matrix Inequality constrained optimization), I solved for the range of such values in the example provided. In any event, you can directly verify my results by computing the eigenvalues of the resulting correlation matrices when values of $corr(x_2,x_3)$ are inserted.

EDIT: There is a case in which transitivity holds. If $corr(x_1,x_2) = corr(x_1,x_3) = 1$ . Then $corr(x_2,x_3)$ must also equal 1.