How to report the performance of clustering algorithm on labeled data?

Question

I have clustering algorithm and I cluster labelled data sets to see my clustering algorithm performs.

I was thinking to report performance of my clustering in sense of some score, like RandIndex score. Idea I had was to compute rand score for each clusters and take the average of it over cluters and report it. Is it faie to do this? Do you know any better way of doing it ?

Here is the table that represent results. Column names shows the classes

     Cluster_Table

  Cluster subtype0 subtype1 subtype2 subtype3 subtype4 subtype5
                1       12        0        0        0        0        0
                2        0        5        0        0       11       10
                3        0        0        9        0        0        0
                4        0       11        0       16        0        0
                5        0        0       16        0        0        0
                6        0        9        0        0        0       15
                7        0        0        0        0       14        0
                8       13        0        0        0        0        0
                9        0        0        0        9        0        0

Answer 1

Yes.

The usual academic (as in: this doesn't work in real life) is to compute a clustering on the data without the labels, then add back the true labels, and use Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) to compare the two panelings.

You can't use "regular" precision and recall, because the clustering sill not give a named class. It will name classes 1,2,3,... so you need ARI, NMI etc. that can compare two results even when the names aren't the same.

Beware of overfitting. A lot of people tweak their algorithm parameters until they get a "good" result, by peeking at the evaluation result. When you then try this method on real - unlabeled - data you often have no idea how to set these parameters.

Answer 2

I believe your question does not have a definite answer, so here are my ideas how you can show the power of your clustering algorithm:

1. Use a bunch of different algorithms on the same data (without the labels) e.g. k-means, hclust, etc.
2. Construct a decision tree on top of the results of each algorithm. (It will be used only for presentational purposes). Visualize the decision tree and inspect carefully the splitting rules. They will give you some ideas on which factors played an important role in the respective clustering algorithm.
3. Create some 2-dimensional plots, while coloring each point acording to each of your algorithms. Some of these plots will help you visually investigate the data further.
4. Do steps 2 and 3 for the actual classes that you have and try to tell which of the previous algorithms works best on your data.

A summary of my answer is that in your case it is better to show your data on a plot instead of using a statistic. When you present your work, others will be able to actually "see" that your algorithm is better than the rest of the clustering algorithms you have tested.

Answer 3

RandIndex is a measure you use when clustering unlabeled data.

• If you want to deploy this clustering algorithm on labeled data, you are basically saying that classification that ignores the labels is better than classification that uses the labels. While that might theoretically be true for some data-sets, your clustering algorithm that is effectively used for classification would then need to be evaluated like a classifier. Use accuracy, F1, AUC or a similar classification performance metric to compare your clustering algorithm's classification performance to the classification performance of proper classifiers.
• If you never want to deploy this clustering algorithm on labeled data, then just don't do it, not now not ever. You would be using it on labeled data now that cannot be representative of the unlabeled data that you want to deploy it on later.