I have a dataset of $N$ variables of which I know that (at least) one combination of their disjoint subsets results in non-negative correlation coefficients for all variables in these subsets. The subsets might have a different number of elements.

For example: The variables $A,B,C,D,E,F$ have the following correlation coefficient matrix (only showing the upper triangle)

    A     B     C     D     E     F   
A   1     0.32  0.3   0.04 -0.03 -0.02
B   0     1     0.3   0.08 -0.03  0.01
C   0     0     1     0.01  0.02  0.02
D   0     0     0     1     0.28  0.3 
E   0     0     0     0     1     0.3 
F   0     0     0     0     0     1   

In this case, a complete description of this data in positive-only correlations would be $\{A,B,C\}$ and $\{D,E,F\}$ (The sub-matrices of these two sets only contain positive correlation coefficients). Another possible combination is $\{A,B,C,D\}$ and $\{E,F\}$ (as is $\{A,B\}, \{C,D\}, \{E,F\}$).

In the example above, $\{A,B,C\}$ and $\{D,E,F\}$ was indeed the looked-for combination, but I cannot find a way to favor this possibility of decomposition without prior knowledge. Since $D$ has a correlation coefficient close to $0$ (and has a higher absolute value with $\{A,B\}$ than with $\{E,F\}$), I don't see using the absolute value of the correlation coefficients as the way to go.

As said, I know that there are such relations in my data, so I'm not trying to find something in the data that isn't there, I'm just trying to retrieve the inherent structure of the data.

Is there a clever way to find such a combination of subsets? If so, is there a generalization if such a decomposition of the data set exists, but does not use all $N$ variables (let's say in the example above, another variabe $G$ would be added which is just noise and thus might -- by chance -- be anti-correlating to any other number of variables)?

I have been looking into ways to do this by constructing a graph where the variables are the nodes and the edges describe only positive correlations. The cliques in this graph then represent all possible decompositions, but not only a complete one.


In general, this problem fits into the scope of community detection in networks, which is difficult to generalize and rather ill-defined. However, there has been some progress focusing on the structure of correlation matrices: Community Detection for Correlation Matrices. Unfortunately, very few methods will give you structure over only a subset of the graph, and I don't know any such methods designed for correlation matrices. Perhaps an extension of link communities or OSLOM would be able to determine those subsets that are statistical significant.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.