We have a large collection (1-2M) of documents of various domains (politics, design, programming, etc.). And lets assume that we don't know the exact number of domains. And our goal is to build relevant topic models for these domains. For example I want to find N topics in each domain. For politics it could be: foreign policy, government, economic policy, etc. For design domain: web design, architecture, etc. And so on.
If I understand correctly, when using a LDA it is necessary to form a training set for specified domain (e.g. politics). But problem arises if there is no information about domain. How can I form a training set in that case? I was thinking about building index, extracting documents, that contain significant words (defined by TF-IDF). But it seems that it is not the best solution.
If instead of LDA we can use HDP and handle training set compiled with random documents. But there arises another problem - a huge vocabulary, which may not fit in the memory (we need much more words to describe several domains, than one).
So the problem is: what is the best way to build topic models (or one topic model?) of various (and unknown) domains? And them apply this models to new (unseen) documents.
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ Please unfold abbreviations "LDA", "TF-IDF", "HDP" in yout question. Not everybody knows these acronyms. $\endgroup$– ttnphnsNov 30, 2012 at 6:45
-
$\begingroup$ @ttnphns, I've added links to wikipedia. $\endgroup$– pepperedNov 30, 2012 at 10:20
-
$\begingroup$ @peppered what did you end up doing for this? $\endgroup$– dendogSep 29, 2020 at 15:43
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
9
Hierarchical topic models are designed precisely for this task. These models infer topic hierarchies where high level topics (poltics, sports) have subtopics beneath them (foreign policy, baseball).
The original formulation is from this paper
And this very recent formulation, which is able to infer more accurate hierarchical structures
-
$\begingroup$ Thanks for your answer! But how can I form a training set (from a huge collection), which would cover all domains? Or it supposed to use all (about 2 millions) documents in original collection? $\endgroup$– pepperedNov 29, 2012 at 21:29
-
$\begingroup$ Yes, the idea is that it will discover the tree structure on it's own from the entire document collection. $\endgroup$– jeradNov 29, 2012 at 21:36
-
$\begingroup$ If I understand correctly for big collections it should be used an online version of hLDA (or nHDP)? $\endgroup$– pepperedNov 29, 2012 at 21:41
-
$\begingroup$ @pepper, I don't think that's the case. What makes you think that? On-line inference is generally more difficult than batch processing because of the constraints on computing time. $\endgroup$– jeradNov 29, 2012 at 21:44
-
1$\begingroup$ @Pepper: Yes, you could also fit topic models recursively as you say. That may or may not be more computationally tractable, I'm not sure, but I suspect the resulting topic hierarchies would be considerably inferior in quality. Inferring the hierarchy all at once, as with hLDA (or nHDPs), will provide the optimal global structure, whereas recursively fitting LDA models to construct a tree would be locally optimal, a sort of greedy algorithm. I'm not expert however, so if this point is important you might make it a new question. $\endgroup$– jeradDec 4, 2012 at 19:19