1
$\begingroup$

I have clustering algorithm and I cluster labelled data sets to see my clustering algorithm performs.

I was thinking to report performance of my clustering in sense of some score, like RandIndex score. Idea I had was to compute rand score for each clusters and take the average of it over cluters and report it. Is it faie to do this? Do you know any better way of doing it ?

Here is the table that represent results. Column names shows the classes

     Cluster_Table

  Cluster subtype0 subtype1 subtype2 subtype3 subtype4 subtype5
                1       12        0        0        0        0        0
                2        0        5        0        0       11       10
                3        0        0        9        0        0        0
                4        0       11        0       16        0        0
                5        0        0       16        0        0        0
                6        0        9        0        0        0       15
                7        0        0        0        0       14        0
                8       13        0        0        0        0        0
                9        0        0        0        9        0        0
$\endgroup$
2
  • $\begingroup$ Did you try a search here? This kind of question must have been asked not once already. $\endgroup$
    – ttnphns
    Nov 1 '17 at 13:43
  • $\begingroup$ Yes, Couldn't find proper answer for that ! $\endgroup$ Nov 1 '17 at 13:45
2
$\begingroup$

Yes.

The usual academic (as in: this doesn't work in real life) is to compute a clustering on the data without the labels, then add back the true labels, and use Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) to compare the two panelings.

You can't use "regular" precision and recall, because the clustering sill not give a named class. It will name classes 1,2,3,... so you need ARI, NMI etc. that can compare two results even when the names aren't the same.

Beware of overfitting. A lot of people tweak their algorithm parameters until they get a "good" result, by peeking at the evaluation result. When you then try this method on real - unlabeled - data you often have no idea how to set these parameters.

$\endgroup$
1
$\begingroup$

I believe your question does not have a definite answer, so here are my ideas how you can show the power of your clustering algorithm:

  1. Use a bunch of different algorithms on the same data (without the labels) e.g. k-means, hclust, etc.
  2. Construct a decision tree on top of the results of each algorithm. (It will be used only for presentational purposes). Visualize the decision tree and inspect carefully the splitting rules. They will give you some ideas on which factors played an important role in the respective clustering algorithm.
  3. Create some 2-dimensional plots, while coloring each point acording to each of your algorithms. Some of these plots will help you visually investigate the data further.
  4. Do steps 2 and 3 for the actual classes that you have and try to tell which of the previous algorithms works best on your data.

A summary of my answer is that in your case it is better to show your data on a plot instead of using a statistic. When you present your work, others will be able to actually "see" that your algorithm is better than the rest of the clustering algorithms you have tested.

$\endgroup$
0
$\begingroup$

RandIndex is a measure you use when clustering unlabeled data.

  • If you want to deploy this clustering algorithm on labeled data, you are basically saying that classification that ignores the labels is better than classification that uses the labels. While that might theoretically be true for some data-sets, your clustering algorithm that is effectively used for classification would then need to be evaluated like a classifier. Use accuracy, F1, AUC or a similar classification performance metric to compare your clustering algorithm's classification performance to the classification performance of proper classifiers.
  • If you never want to deploy this clustering algorithm on labeled data, then just don't do it, not now not ever. You would be using it on labeled data now that cannot be representative of the unlabeled data that you want to deploy it on later.
$\endgroup$
8
  • $\begingroup$ Sure, i got your point, but I would like to report people how good is it work. that's why i tested it on ground truth data and ofcourse I can show it visually, but I was thinking of proper metric that is representative ... $\endgroup$ Nov 1 '17 at 13:39
  • $\begingroup$ You would be telling people "look how good it is at doing X, therefore we should use it to do Y" $\endgroup$ Nov 1 '17 at 13:42
  • $\begingroup$ Well, I have experiments on multiple setting and it's really hard to put all in visual manner... Also I need to have quantitative measure of goodness ! $\endgroup$ Nov 1 '17 at 13:44
  • $\begingroup$ Of course, that is why RandIndex and similar clustering performance metrics have been developed for unsupervised learning. If in your application scenario you would cluster on unlabeled data and you would come to know the true labels later, then you can evaluate your clustering like a classifier with classification performance metrics. If not, then don't test it on supervised data-sets which would be per definition not representative of your application scenario. $\endgroup$ Nov 1 '17 at 13:47
  • $\begingroup$ The Rand index compares twox independent labels. Usually, one is obtained from clustering, and one is the "ground truth" classification labels. So it's *not used with unlabeled data - because you need a second labeling. $\endgroup$ Nov 2 '17 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.