First of all, how to know wheter document clustering result was "good" or "bad"? Not in ordinary k-means, but in algorithm that enable automatic cluster number, like g-Means. Came across Q&A related to cluster evaluation, I found that I can implement F-measure to determine whether system giving "good" or "bad" result regarding to clustering. Case is : you want to cluster documents. If it's you that define number of cluster, say k = 5, then it's easy to find True Positive (TP), TN, FT, and FN. But what if my automatic system say there is k = 7, because you have some outlier documents that 'mathematically' don't match with any clusters, so the system create new cluster, thus adding number of clusters. Let's say system assign some outlier documents to cluster 6 and the rest to cluster 7. Since document relevance is very subjective to human, the human tester can say that outlier document belongs to cluster 5. If I use F-measure, This testing result could give my sistem very bad result (accuracy). Is there any other possible method? Thanks for your answer.
You can only compute the F measure if you have labeled data. But then you should be using classification, not clustering.
When using clustering, any use of labels is questionable. Because you say "the best algorithm produces what I already know", but then you don't need to run it at all! Whereas a clustering tjat e.g. found clusters within your labels (probably a much better clustering result) will perform bad on your measure!