Quantitative results of cluster analysis

Question

Currently, I am doing a clustering for two sets of data. One smaller dataset (about 100 data) got ground truth labels, and one larger dataset (about 2000 data) has no ground truth labels.

For the smaller dataset, obviously, I can obtain quantitative results like accuracy, sensitivity and specificity.

However, for the larger dataset, I have no ground truth and couldn't get any useful quantitative results.

1. The only thing I found useful is the 'mean silhouette value', which can measure the cluster performance. However, it based on some distance measure that can only tell people how separate are the clusters. I am wondering if there are other 'better' or 'more appropriate' quantitative analysis for data without labels.

2. Because the data are without labels, I am also wondering if we can somehow have a 'uncertainty' measure about the clustering results like how confident about the cluster results?

3. For the smaller dataset with labels, except accuracy, sensitivity and specificity, any other quantitative results I can get? For the classification algorithm, we can do a cross-validation, is there any method we can use to do such a cross-validation for clustering? Also, can we get ROC analysis for clustering task?

Answer 1

How are the data sets related? IF both data sets are drawn from the same distribution (they describe the same problem) than you can use the labeled set as a "test set" for the clustering. Basically you treat the clustering algorithm as a classifier. The only problem is that you must find a match between the output of the clustering algorithm and the actual labels.

You might use some simple matching (ex: instances labeled GREEN are more often clustered in cluster 2 and BLUE in cluster 1 so cluster 1== BLUE and cluster 2 == GREEN).

More elegantly you can compute the Mutual Information between the clustering output and actual labels. Mutual Information has a nice property, that one doesn't need to know the exact matching. MI will give high scores if most of the matching are consistent. Think of it as a correlation coefficient between (cluster <-> actual label) relation.

Also check http://en.wikipedia.org/wiki/Cluster_analysis for some measures. The key phrase there is:

[...] clustering results are evaluated based on data that was not used for clustering, such as known class labels and external benchmarks. Such benchmarks consist of a set of pre-classified items, and these sets are often created by human (experts). Thus, the benchmark sets can be thought of as a gold standard for evaluation.

For ROC usually one needs some "a posteriori" probability, outputted by the classifier, but in your case, the distance between the instance and the cluster center will work. Keep in mind that ROC is computed for a specific label at a time (i.e. one vs all). So for 5 labels you will get 4 independent AUROC values.

IMHO I strongly advise yo to do the CV for clustering if you have labeled data! Iterate it several times and use the mean of your measure as the performance.

I would also try this: Use some percent (66% usually) of unlabeled data to perform clustering, measure performance using labeled data, repeat the experiment with different randomization (usually 5-10 times) and report mean performance. Unfortunately I don't know if this method will give a good estimate of your real performance. Is it possible that will overfit the labeled data set. This is not a textbook approach, so, use it with caution.