Timeline for Data set for document topic discovery
Current License: CC BY-SA 3.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 3, 2016 at 14:45 | comment | added | roundsquare | @H.Z. I'm a bit confused... do you mean that each document has a unique topic (in your example, there are a million distinct topics) or that there a large number of topics (maybe 100 or 1,000) and that each document has some mixture of the topics? In the latter case, LDA will work (albeit, slowly unless you use the online version as xeon suggested). | |
| Apr 26, 2016 at 4:11 | comment | added | Vladislavs Dovgalecs | @H.Z. You should use then online versions of the LDA algorithm like the one implemented in Vowpal Wabbit. It will happily crunch terabyte scale dataset even on a modest laptop. | |
| Jun 25, 2015 at 18:08 | comment | added | wij | You should update your question to reflect this. I did not realize you are considering a big data scenario. | |
| Jun 25, 2015 at 15:02 | comment | added | HHH | I am familiar with LDA. It is useful when the documents are talking about the same topic and we would like to know what that topic(s) is. However consider a case in which you got a million document and each document is talking about a unique topic. There is no way to use LDA in this scenario. In this scenario we should first do a clustering to find similar documents and then do a LDA for each cluster which may not be possible when data set is big! | |
| Jun 25, 2015 at 11:52 | history | answered | wij | CC BY-SA 3.0 |