Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I'm an enthusiastic single developer working on a small start-up idea. I reduced a corpus of mine to an LSA/LDA vector space using gensim. Now I have a bunch of topics hanging around and I am not sure how to cluster the corpus documents. I see that some people use k-means to cluster the topics. Can someone please elaborate?

So i had some to properly read up LDA/LSA and took a look at the gensim source. i did not realize that the Similarity Matrix was actually an MXM matrix where M is the number of documents in my corpus, i thought it was MXN where N is the number of topics. Using the matrix as an input to scikit's linkage function, i was able to create a heirachical cluster. Well, i was able to do it for a 1000 docs in my corpus (need more ram to handle a large corpus).

share|improve this question
1  
You can treat the topic mixture vector for each document as its position in this latent topic space. Simply run your clustering using this as the input data. What is the point of the hard clustering though? The topic mixtures already give you a lot of information about how documents are similar/different. – Nick May 22 '12 at 20:36
Would you kindly provide more details on how you did it? That will be a great help. Best regards, – Dibyendu Oct 3 '12 at 21:52
i actually have to take a look at the code again, the project is currently in hiatus. I will take a look later on today. – osilocks Oct 4 '12 at 10:11

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.