Data is organized in the following way: Group 1 of items A, B, C, group 2 of items D, E, F. Each item has an associate value, let's assume A(1), B(2), C(3), D(10), E(11), F(12). I want to show/ask whether items within a group are more correlated with each other than with items outside the group. This reminds me of autocorrelation in space/time like Moran's I. I thought to generate two columns for a single Pearson correlation, comparing each item to other items within its group, i.e. A:B, A:C, B:A, B:C, C:A, C:B, D:E, D:F, E:D, E:F, F:D, F:E. I'm not concerned with the influence of group size on the group's 'weight' but maybe there are other issues.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Let's start by assuming that you have a solid notation of what "correlation" means in your context. It sounds like each item is representable as a vector and you've chosen the Pearson correlation coefficient as your metric of correlation -- which is totally reasonable. You do mention measuring the correlation of A:B and of B:A which would be the same, so maybe I've misunderstood your question.
Nevertheless, if that setup is correct, then you can compute a matrix of correlations -- each row and column represents an item (A, B, C, D, E or F) and the entries are your measured Pearson correlation coefficients. You can then interpret the correlation matrix as a graph. The matrix is symmetric so the graph is undirected. You then want to partition the graph into two subgraphs in order to maximize the connections within subgraphs and minimize connections between subgraphs. This is known as the "minimum-cut problem" and there are algorithms, such as the Stoer–Wagner algorithm, that can solve it.
An issue here might be that the min-cut separates the graph into one item in one group and all the rest in another group. You might want to try to constrain it to have groups of equal size.
If your matrix is small (e.g., has only six rows/columns as you describe in your setup) then you might be able to get away with a brute force approach and simply try all possible item-to-group allocations and measure the between group correlations for each. The between group correlations would be the sum of the absolute values of the correlations between each item in each group. (Or, possibly, the root mean square of the correlations.)
A different approach, would be to assign each item to a group and compute the first principal component of each group, call them P_1 and P_2. Then, for each item, you compare correlation of that item to P_1 and and that item to P_2 and reassign the item to whichever shows the higher correlation. If you reassignments match your original assignment then you conclude that "items within a group are more correlated with each other than with items outside the group" as hoped for.
