Validating clustering results with labeled data I am working on a clustering algorithm and would like to validate its performance against a well-known and used dataset: the KDD-CUP 99 dataset (http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html). With this dataset, both unlabeled and labeled test data is provided. My question is, how should I validate my clustering algorithm's performance? 
Let's say the results of my algorithm are as follows: 
x1 -> cluster A 
x2 -> cluster A 
x3 -> cluster B 
x4 -> cluster A 
And let's say the labels provided are as follows: 
x1 -> cluster 1 
x2 -> cluster 1 
x3 -> cluster 1 
x4 -> cluster 2 
Given that the cluster labels are completely different, how should I compare these? In this case, an obvious assumption would be to say that cluster A is probably the same as cluster 1, but this may not always be this obvious. Is there any standardized way to evaluate such situations? 
 A: Look into distances between clusterings. They all use what is called the confusion matrix between two clusterings. Well known are the Rand index and the adjusted Rand index, but I generally recommend using either Variation of Information or the not well known split-join distance (see e.g. Comparing clusterings: Rand Index vs Variation of Information and How to interpret these indices/metrics for comparing partitions intuitively out of these images? for more discussion).
A: Be really careful with this data set.
KDD Cup '99 dataset (Network Intrusion) considered harmful


*

*This data set does is no way resemble current network traffic. Assuming that it would indicate usefulness for detecting network intrusions is foolish.

*With well-crafted methods (such as some simple IP filters), most of the attacks present in this data set can trivially be detected. On the raw data, a simple TTL is even vs. TTL is odd filter apparently is able to achieve 100% correct.

*The data set has a massive amount of duplicates. If you do naive cross-validation, your results are likely overfitting, because you have duplicates in test and training sets.

*This is a classification data set, not a clustering data set. Clusters and classes are not the same thing. With clustering you want to discover something new in you data, and using classification labels you actually punish if anything new was discovered...

*Attributes are categorial, binary, false numerical (IP) - there are next to no continuous attributes in this data sets. Most distances on this data set are entirely meaningless.
All in all, stay away from this data set.
