Quantifying how successful results of clustering are with a test I have a three types of experiments with many measurements taken. I want to show that using clustering on the unlabelled measurements, I can uncover the three groups. Now using kmeans plus, I get around 60% accuracy. Which if it was binary, would be poor, but how do I quantify how good that is for three categories? Is there a p value I can compute and compare against? I am thinking that if there were 100 categories 60% for the correct labelling is great and far from random, so how can I do this with a test?
 A: You can get a p-value from a Monte Carlo resampling approach.
Take your number of obserations (n) and randomly distribute them into k groups. Repeat this procedure many times (10,000+). Count the number of times in which this process yields a clustering equal to or better than your own. Put that number over the total number of Monte Carlo trials, and that's your p-value: the chance of getting a clustering as good or better than yours by chance alone.
I agree with Greg Snow, though, that a p-value isn't a measure of prediction strength. Reporting the p-value along with Cohen's kappa and (of course) the descriptive statistics would probably be a good idea.
A: You could use Cohen's kappa test for the 3 x 3 classification table (actual membership by clustering membership).
A: Testing for a statistically significant relationship and quantifying the quality of predictions are 2 very different proceedures.  With enough data it is possible to show something is statistically significant even when the effect is to small to care about.  For your example with 3 groups (assuming they are the same size) we would expect a  completely random clasiffication to get the right answer $\frac13$ of the time.  A method that was correct 35% of the time could be found to have a very small p-value, but would not be a practical improvement above 33%.  On the other hand, with small data you may have a classifier that is right 90% of the time but not statistically significant.
