Benchmarking hard clustering results against soft clustering results (ground truth)?

Question

Well, I don't have labels (ground truth) for my data points. However, by domain knowledge, I am certain that a particular soft clustering algorithm will produce satisfactory results. Hence, I may use its output, denoted by C, as a surrogate of the ground truth. Let's say $$C=\{(0.1, 0.9), (0.2, 0.8, \ldots)\},$$ where $C_1=(0.1, 0.9)$ means that the 1st data point is in Cluster 1 with 0.1 probability and in Cluster 2 with 0.9 probability.

Now, I want to benchmark some other hard clustering algorithms against my "ground truth", $C$.

What is the rightful way of doing so? Is there an already-formed convention for this?

Answer 1

Most "external" (supervised) evaluation measures such as ARI should work, if you take element weights into account. Most are based on an agreement of pairs; and you may argue that a pair exists with $p_1\cdot p_2$ in your soft clustering, if $p_1$ and $p_2$ are the probabilities of objects 1 and 2.

They probably just become much more computationally intensive, as you can't just use the cluster intersection sizes anymore.