Cluster analysis with interchangeable, binary classes

Question

The Data:

I have the results of an experiment where participants were given 30 stimuli and asked to sort them into two groups. The participants were asked to sort them into the two groups without any information about what the groups should be or how they should be sorted.

There is data (see 1) from 93 participants. Each column represents one stimulus {S1, S2,...} and the cells hold a binary value {1, 2} that represents the group it has been put into.

So for example: Participant 1 put {S1, S2, S3, S4, S5} into group 1, whereas participant 2 put {S1, S2, S5} into group 2 and {S3, S4} into group 1.

The Goal:

I want to find out whether some stimuli were grouped together more often than others, i.e. whether participants found similar groups.

The Problem:

The groups {1, 2} are interchangeable.

So, for example, participant A put S1, S2 and S3 into group 1, while participant B put these three into group 2. They both would have grouped the stimuli together and I would want to see that these stimuli have been grouped together more often, even though into different groups.

It does not matter whether they were assigned group 1 or group 2. It is ony important which stimuli are in the same group.

How could I get these relations out of my data?

Similar Problem:

The problem mentioned here is very similar to this one. However, the group labels don't change between participants (i.e. 1 is always "disease present" and 0 is always "disease not present"). Something like the suggested hierarchical clustering looks very promising, but I am not sure how I could make it account for the changing group labels.

Answer 1

Your task is the comparison of partitions without knowing any labels for the groups. This task is also called "assessment of agreement between clusterings" (contrasted to "assessment of agreement between classifications" - where we know the class labels).

You have not done any clustering yet, but admit that your participants were that clustering machines who gave you, each, a vector of 2-cluster solutions. Each solution is a row (a participant) in your data.

Actually, the number of clusters might have been more than two and even different for different participants.

Cluster solutions, the participants, are similar if they partitioned the set of stimuli similarly. For example, two solutions A and B are similar if there are many stimuli out of the 30 which tend to encounter in one some cluster in A solution and one some cluster in B solution; while stimuli found in different clusters in A tend to dissociate in B too. You should compute a measure of clustering agreement (also known as external clustering criterion) between each two solutions (participants).

Upon obtaining such square symmetric 93x93 similarity matrix, you can perform cluster analysis on it, to cluster the participants. A hierarchical cluster analysis would be the most appropriate here. It will cluster your participants into groups of similar ones: who has partitioned the stimuli similarly. Then you will turn back to the data to reveal the patterns characteristic for each such group.

Clustering agreement measures are multiple. To list some: Dice (F1), Ochiai (Folkes–Mallows), Adjusted Rand (Cohen's kappa), Phi correlation (Hubert's Gamma). These are called "pairs" aka "comembership confusion matrix" measures. Another line of measures to use include Overlap, F Clustering Accuracy, Mutual Information, and Homogeneity&Completeness V. On my page, download collection "Compare partitions" to read about these and related measures. If you are an SPSS user, you can use the macro !CLUAGREE I wrote to compute the square matrix of any such measure, which you will then submit to a cluster analysis. (You will have first to transpose your data, to make the participants - the "cluster solutions" - the columns rather than the rows.)