3
$\begingroup$

In an 'unguided' experiment, we asked several people to make groups with a defined number of objects presented to them. They were free to do as many classes as they wanted, and to choose their names, but they had to detail precisely the rationale of their groupings.

Their detailed explanations were very useful to us, but we also wonder if there are ways to compare their classifications. There are obvious difficulties: - all classifications have different numbers of classes; - the labels given are also different (the class 'A' of a given observer may correspond to the class '5' of another user).

Even if it seems difficult to consider, is there some mathematical tools or indices to study the 'conformity' between those classifications, and to find which ones are the most similar and the most different? ('Being similar' could mean that most of the time, the objects of two given classifications are in the same classes, even if those classes have not the same name.)

NB: These objects were various kind of silex, and most people classified them according to color, texture/roughness, etc. But in fact, the essential point is that we have for each object (each silex) something like: user 1 classified it in cluster 'A', user 2 classified it in cluster 'red', user 3 classified it in cluster '5', etc. And mathematically, the idea is really to compare a list of cluster attributions between various users.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Can you post a small example dataset for people to work with? How many objects were there? How many participants were there? $\endgroup$
    – gung - Reinstate Monica
    Commented Dec 21, 2018 at 12:06
  • $\begingroup$ "the idea is really to compare a list of cluster attributions between various users." Does this mean you want to compare overall paritionings (e.g. the total assignment of objects to clusters made by person 1 vs. person 2). Or, do you want to compare individual clusters (e.g. person 1's "class A" vs. person 2's "class 5")? $\endgroup$
    – user20160
    Commented Dec 23, 2018 at 1:50
  • $\begingroup$ Please post a small example dataset for people to work with. How many objects were there? How many participants were there? $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 2, 2019 at 13:45

3 Answers 3

Reset to default
2
$\begingroup$

My first thought would be something called formal concept analysis, a textbook is here.

I have no experience with this multivariate technique, there is not much information on this site (you can try to search), and there is no R package ... but here is a page including links to software.

Here is, more or less, what you would have to do with your data:

  1. For each user and each object, go through the users explication of his choice of category, and take note of the words used in the explication: red, blue, rough, matte, itchy, ... and so on (when taken the union of this lists, probably long)

  2. Then make a data table, with rows for objects and columns for description words. You would have one such table for each person. Put 1's in the cells where this word was used.

  3. This will be the input data for the formal concept analysis (FCA). The output will be some hierarchical clustering of the objects.

  4. Now you can compare these formally derived groups with each persons group.

Other ideas are certainly possible, and as I have no experience with FCA, this is just a fast idea, but it would be interesting to try ...

Improve this answer
$\endgroup$
1
$\begingroup$

Clustering comparison measures may be useful.

In particular Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI ; or even better, Adjusted Mutual Information, AMI) are worth trying. They can handle different numbers of clusters, and they measure how much more than random two partitionings agree.

Improve this answer
$\endgroup$
1
  • $\begingroup$ +1 to remind readers: don't use NMI. Several works (e.g. this and this) have shown how the measure can mislead users in subtle ways. $\endgroup$
    – Arya McCarthy
    Commented Mar 24, 2021 at 23:50
1
$\begingroup$

The problem you have is not "Classification". The technical term which is used for this situation is "Clustering". In clustering, the labels are unimportant since there is no ground truth to compare with. Only the position of an element matters. For instance, assume this case:

1112223334
AAABBBCCCD

These two cases are completely similar from the clustering point of view.

I have no idea how to exactly match them but it is what I may do for this problem (not 100% correct):

I would do a pairwise clustering comparison.

  1. I pick an external clustering validation index (i.e., Adjusted Rand Index)

  2. I choose the idea of person $i$ as ground truth.

  3. I choose the idea of person $j$ as predicted clustering.

  4. Using the Adjusted Rand Index, I will validate the idea of person $j$ based on the idea of person $i$. If they match, it will give me a high value, otherwise, it should be small.

  5. I create a similarity matrix based on steps $2$ to $4$ for all pairs.

  6. After finding $n \times n$ similarity matrix when $n$ is the number of voters, using Hierarchical clustering and Ward’s method (or $k$-medoids), I will try to find the most similar ideas and cluster them in a new cluster.

This is what I may do for this problem but probably there are better ways to solve the problem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.