3
$\begingroup$

The word2vec tool uses deep learning to compute vector representations of words. They've mentioned that - "The word vectors can be also used for deriving word classes from huge data sets. This is achieved by performing K-means clustering on top of the word vectors."

When I execute the code, it seems that the tool computes word vectors from a dataset and clusters it into 200 classes. What if I wanted to use another method of clustering? How would I measure how 'good' these classes are, or how well the clustering algorithm is working?

Also, it seems to me that the word vector representation would be a very sparse vector space. Couldn't the clustering be improved by PCA or some method of dimension reduction? Again, how can I test how well the clustering is performing? If I can figure out a way to evaluate, I can change the code and add PCA and see if it does better.

EDIT - I found some papers that mention that word clustering can be used in document classification. The idea is to use the word clusters instead of a bag-of-words as features for classification, and this reduces the feature space while preserving 'redundant' features. I'm looking more into this, but it seems like I could use the word clusters on a document classification task and see if it performs better if I change the clustering algorithm. Would this work?

A follow-up question - how would I implement this approach of using word clusters as features?

Improve this question
$\endgroup$
4
  • $\begingroup$ This is a good question, but note that 'how good are the clusters?' requires some notion of what the point of the clustering is. What is your goal here? What would it mean to you for a particular clustering to be good or bad? $\endgroup$
    – gung - Reinstate Monica
    Apr 15, 2014 at 16:11
  • $\begingroup$ @gung, that's part of what I'm trying to figure out. Sorry if the question is unclear. I can't think of an application of clustering word vectors. If I could think of a good application, then I could figure out how to evaluate it. If I implemented a different way of clustering these word vectors, how could I compare with the original code? $\endgroup$
    – devikad
    Apr 15, 2014 at 16:23
  • $\begingroup$ I sympathize, @devikad, but it is hard to know whether a clustering helps achieve your goal w/o knowing what your goal is. FWIW, some common goals for clustering are to facilitate storage & retrieval of data w/o regard for whether the clusters represent anything meaningful, & to make a guess about real (but latent) underlying classes that form an important part of the data generating process. $\endgroup$
    – gung - Reinstate Monica
    Apr 15, 2014 at 16:27
  • $\begingroup$ @gung, thanks for the pointers, I'm looking into those applications. I've edited my original question - what do you think? $\endgroup$
    – devikad
    Apr 15, 2014 at 16:41

2 Answers 2

Reset to default
1
$\begingroup$

Knowing whether a cluster is "good" is an open ended problem to which you can't really give a simple answer. But, you can always check whether the clusters you get from your black box are stable.

Here's what I'd try: Divide your training sentences/documents into two halves, and run your clustering black box separately on each half. Then, check whether the clusters obtained on each half are "similar" (e.g., check the fraction of word pairs that are in the same cluster both times vs. pairs that go from being in the same cluster to being separated). If the clusterings you get on different halves of the data are similar, then at least your black box is fitting some kind of stable signal. On the other hand, if the two clusterings are completely different, then it may be that the black box is effectively just fitting to noise.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the suggestion. This seems like something that can yield some results. Could you explain further what you mean by 'fitting a stable signal' vs. 'fitting to noise'? $\endgroup$
    – devikad
    Apr 15, 2014 at 18:18
  • $\begingroup$ The point I was trying to get at is that it's important know whether the clustering is driven by "persistent" or "transient" aspects of the training data. In your problem, you probably want your clustering to reflect something about (english) language in general, and not something about, say, Reuters news clippings in 1998. If your clustering algorithm gives very different results on different halves of the training data, then the algorithm is probably fitting to transient details of the training set that will not appear in future datasets, which may just as well be treated as noise. $\endgroup$
    – Stefan Wager
    Apr 15, 2014 at 19:01
  • $\begingroup$ thanks for explaining. That makes sense. I think I'll start with this approach and see what I get. $\endgroup$
    – devikad
    Apr 15, 2014 at 20:27
1
$\begingroup$

It's really hard to compare clusters and know which one is better without having an application in mind. Part of the problem in your case is going to be that most words in natural language are rare. This means that their clustering will be unreliable but it also means that it doesn't matter much how you cluster them because they are improbably anyway.

If you're going to be doing document classification it is not obvious to me that you would get an advantage by doing the clustering first rather than keeping the continuous space representation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.