I have a huge data set and want to cluster the objects in there with my own clustering algorithm.

I also have a few lists of objects I know belong to the same cluster.

The problem is, that none of these lists depicts a complete cluster and the number of lists I have is very limited in comparison to the number of objects and the number of clusters I expect to find. Another problem is, that I am far from being an expert in both the clustering as well as the statistics domain.

My question now is: Can I use my lists as "ground truth" data to evaluate my algorithm in any way that would make me comparable to others and how would I do that (what scores could I calculate in order to compare myself)?

An approach I allready thought of is the following:

  1. From my lists I can form pairs, that should be in the same cluster ([A,B,C] -> [A,B],[A,C],[B,C])
  2. I run my algorithm and count how many pairs are in the same cluster (A and B are in one cluster, C is in another cluster)

So [A,B] was found correctly and [A,C] and [B,C] were not, which gives me a score of 1/3. This test could be repeated with different subsets of my data set, each time evaluating how many of the pairs where found and how many were not.

At the moment this is the only test I can think of, but I don't think this will help me with being comparable to other approaches.


1 Answer 1


If the lists are disjoint and refer to different clusters, then you can trivially use the existing measures.

Simply subset your data after clustering to contain only the points (and their cluster labels) where you have true labels.

Note that this gives the approach that you "thought of" if you use any of the standard pair counting metrics such as ARI.

  • $\begingroup$ "sunset your data"? $\endgroup$ Jun 26, 2018 at 21:58
  • $\begingroup$ Seems legit, but this will evaluate clustering of ~300 million data points based on just a few tens of points. Wouldn't this be problematic? $\endgroup$
    – larsman
    Jun 27, 2018 at 7:14
  • $\begingroup$ It only evaluates the labeled points. It doesn't evaluate the entire data because you don't have labels. It's the best you can do, isn't it? $\endgroup$ Jun 27, 2018 at 7:16
  • $\begingroup$ The evaluation will still measure how much the entire clustering agrees with your labels. $\endgroup$ Jun 27, 2018 at 7:17
  • $\begingroup$ This is not the answer I have hoped for, but you are probably right. Thank you very much! And thanks for the hint towards ARI ;) $\endgroup$
    – larsman
    Jun 27, 2018 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.