How can I evaluate the performance of a system that generates word clusters? The word2vec tool uses deep learning to compute vector representations of words. They've mentioned that - "The word vectors can be also used for deriving word classes from huge data sets. This is achieved by performing K-means clustering on top of the word vectors."
When I execute the code, it seems that the tool computes word vectors from a dataset and clusters it into 200 classes. What if I wanted to use another method of clustering? How would I measure how 'good' these classes are, or how well the clustering algorithm is working?
Also, it seems to me that the word vector representation would be a very sparse vector space. Couldn't the clustering be improved by PCA or some method of dimension reduction? Again, how can I test how well the clustering is performing? If I can figure out a way to evaluate, I can change the code and add PCA and see if it does better.
EDIT - I found some papers that mention that word clustering can be used in document classification. The idea is to use the word clusters instead of a bag-of-words as features for classification, and this reduces the feature space while preserving 'redundant' features. I'm looking more into this, but it seems like I could use the word clusters on a document classification task and see if it performs better if I change the clustering algorithm. Would this work? 
A follow-up question - how would I implement this approach of using word clusters as features? 
 A: Knowing whether a cluster is "good" is an open ended problem to which you can't really give a simple answer. But, you can always check whether the clusters you get from your black box are stable.
Here's what I'd try: Divide your training sentences/documents into two halves, and run your clustering black box separately on each half. Then, check whether the clusters obtained on each half are "similar" (e.g., check the fraction of word pairs that are in the same cluster both times vs. pairs that go from being in the same cluster to being separated). If the clusterings you get on different halves of the data are similar, then at least your black box is fitting some kind of stable signal. On the other hand, if the two clusterings are completely different, then it may be that the black box is effectively just fitting to noise.
A: It's really hard to compare clusters and know which one is better without having an application in mind. Part of the problem in your case is going to be that most words in natural language are rare. This means that their clustering will be unreliable but it also means that it doesn't matter much how you cluster them because they are improbably anyway.
If you're going to be doing document classification it is not obvious to me that you would get an advantage by doing the clustering first rather than keeping the continuous space representation.
