Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

My objective is to implement a topic model for a large number of documents (20M or 30M). Let us assume that the number of topics is fixed at 50.

I think implementing an LDA for the above problem would not be difficult. However, I have yet to find an answer for an NMF model. I have read that it is NOT easy to implement an NMF model for a large number of documents.

Is it really not possible to implement an NMF model for my problem?

share|improve this question
3  
I suppose NMF=non-negative matrix factorisation and LDA=latent Dirichlet allocation? It is usually a good idea to explain what abbreviations mean. – Momo Oct 11 '12 at 11:05
Thanks a lot, Momo. Any suggestions? – Dibyendu Oct 11 '12 at 17:31
I don't know the answer, sorry. Maybe try something distributed like research.microsoft.com/pubs/119077/DNMF.pdf – Momo Oct 11 '12 at 19:20
Thanks a lot, Momo. – Dibyendu Oct 12 '12 at 10:53

1 Answer

up vote 1 down vote

Note on implementing LDA for this problem: there are well-designed inference algorithms for huge numbers of documents. Specifically, you should check out "Online LDA", which can adaptively train the topics looking at small chunks of documents at a time.

Paper: http://www.cs.princeton.edu/~blei/papers/HoffmanBleiBach2010b.pdf

Matt Hoffman has python code available: http://www.cs.princeton.edu/~blei/topicmodeling.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.