We have a large collection (1-2M) of documents of various domains (politics, design, programming, etc.). And lets assume that we don't know the exact number of domains. And our goal is to build relevant topic models for these domains. For example I want to find N topics in each domain. For politics it could be: foreign policy, government, economic policy, etc. For design domain: web design, architecture, etc. And so on.

If I understand correctly, when using a LDA it is necessary to form a training set for specified domain (e.g. politics). But problem arises if there is no information about domain. How can I form a training set in that case? I was thinking about building index, extracting documents, that contain significant words (defined by TF-IDF). But it seems that it is not the best solution.

If instead of LDA we can use HDP and handle training set compiled with random documents. But there arises another problem - a huge vocabulary, which may not fit in the memory (we need much more words to describe several domains, than one).

So the problem is: what is the best way to build topic models (or one topic model?) of various (and unknown) domains? And them apply this models to new (unseen) documents.

  • 1
    $\begingroup$ Please unfold abbreviations "LDA", "TF-IDF", "HDP" in yout question. Not everybody knows these acronyms. $\endgroup$ – ttnphns Nov 30 '12 at 6:45
  • $\begingroup$ @ttnphns, I've added links to wikipedia. $\endgroup$ – peppered Nov 30 '12 at 10:20
  • $\begingroup$ @peppered what did you end up doing for this? $\endgroup$ – dendog Sep 29 '20 at 15:43

Hierarchical topic models are designed precisely for this task. These models infer topic hierarchies where high level topics (poltics, sports) have subtopics beneath them (foreign policy, baseball).

The original formulation is from this paper

And this very recent formulation, which is able to infer more accurate hierarchical structures

  • $\begingroup$ Thanks for your answer! But how can I form a training set (from a huge collection), which would cover all domains? Or it supposed to use all (about 2 millions) documents in original collection? $\endgroup$ – peppered Nov 29 '12 at 21:29
  • $\begingroup$ Yes, the idea is that it will discover the tree structure on it's own from the entire document collection. $\endgroup$ – jerad Nov 29 '12 at 21:36
  • $\begingroup$ If I understand correctly for big collections it should be used an online version of hLDA (or nHDP)? $\endgroup$ – peppered Nov 29 '12 at 21:41
  • $\begingroup$ @pepper, I don't think that's the case. What makes you think that? On-line inference is generally more difficult than batch processing because of the constraints on computing time. $\endgroup$ – jerad Nov 29 '12 at 21:44
  • 1
    $\begingroup$ @Pepper: Yes, you could also fit topic models recursively as you say. That may or may not be more computationally tractable, I'm not sure, but I suspect the resulting topic hierarchies would be considerably inferior in quality. Inferring the hierarchy all at once, as with hLDA (or nHDPs), will provide the optimal global structure, whereas recursively fitting LDA models to construct a tree would be locally optimal, a sort of greedy algorithm. I'm not expert however, so if this point is important you might make it a new question. $\endgroup$ – jerad Dec 4 '12 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.