15
$\begingroup$

I have used LDA on a corpus of documents and found some topics. The output of my code is two matrices containing probabilities; one doc-topic probabilities and the other word-topic probabilities. But I actually don't know how to use these results to predict the topic of a new document. I am using Gibbs sampling. Does anyone know how? thanks

$\endgroup$
  • 1
    $\begingroup$ What do you mean by "predicting the topic of a new document"? Do you want to find which one single topic this document was generated from? Do you want to find a mixture of topics for the document? Do you want to label each word in the new document by the topic that word came from? $\endgroup$ – SheldonCooper Apr 11 '11 at 18:17
  • $\begingroup$ Do you want to find which one single topic this document was generated from? Do you want to find a mixture of topics for the document?- I want to answer both of these questions actually...but my corpus is really big, so I cannot afford to re-train all of my model every time a new document is examined $\endgroup$ – Hossein Apr 12 '11 at 8:43
21
+50
$\begingroup$

I'd try 'folding in'. This refers to taking one new document, adding it to the corpus, and then running Gibbs sampling just on the words in that new document, keeping the topic assignments of the old documents the same. This usually converges fast (maybe 5-10-20 iterations), and you don't need to sample your old corpus, so it also runs fast. At the end you will have the topic assignment for every word in the new document. This will give you the distribution of topics in that document.

In your Gibbs sampler, you probably have something similar to the following code:

// This will initialize the matrices of counts, N_tw (topic-word matrix) and N_dt (document-topic matrix)
for doc = 1 to N_Documents
    for token = 1 to N_Tokens_In_Document
       Assign current token to a random topic, updating the count matrices
    end
end

// This will do the Gibbs sampling
for doc = 1 to N_Documents
    for token = 1 to N_Tokens_In_Document
       Compute probability of current token being assigned to each topic
       Sample a topic from this distribution
       Assign the token to the new topic, updating the count matrices
    end
end

Folding-in is the same, except you start with the existing matrices, add the new document's tokens to them, and do the sampling for only the new tokens. I.e.:

Start with the N_tw and N_dt matrices from the previous step

// This will update the count matrices for folding-in
for token = 1 to N_Tokens_In_New_Document
   Assign current token to a random topic, updating the count matrices
end

// This will do the folding-in by Gibbs sampling
for token = 1 to N_Tokens_In_New_Document
   Compute probability of current token being assigned to each topic
   Sample a topic from this distribution
   Assign the token to the new topic, updating the count matrices
end

If you do standard LDA, it is unlikely that an entire document was generated by one topic. So I don't know how useful it is to compute the probability of the document under one topic. But if you still wanted to do it, it's easy. From the two matrices you get you can compute $p^i_w$, the probability of word $w$ in topic $i$. Take your new document; suppose the $j$'th word is $w_j$. The words are independent given the topic, so the probability is just $$\prod_j p^i_{w_j}$$ (note that you will probably need to compute it in log space).

$\endgroup$
  • $\begingroup$ thanks for you answer. I read some stuff about but still a bit confused about "folding in". You are saying that I should keep the topic assignments of the old documents the same, this means that the word-topics assignments should be re-calculated? Is it possible for you to give me a more detailed steps of what should be done? or maybe referring me to a paper or a link that can actually help me clarify this "folding in" process. My first option is to do "folding in". If unsuccessful I will go for the second method you proposed(not sure how well it works comparinfg to folding in).thanks. $\endgroup$ – Hossein Apr 13 '11 at 14:34
  • $\begingroup$ @SheldonCooper: If I understand you correctly, then I doubt that this is the way how to do it: What you do is as if you measure the performance of an SVM on a new test sample by giving the optimization algorithm a few more steps from the current solution including the test sample and then evaluate it on this sample... but: in machine learning you may never test on your training set... and by including the test sample into the model you do exactly that: test on a training sample... $\endgroup$ – Fabian Werner Apr 1 '16 at 14:18
  • $\begingroup$ @FabianWerner I believe the solution did not update the word-topic matrix from the original training. It just re-runs a Gibbs sampler starting with the trained word-topic matrix, and creates a new document-topic matrix. At any rate, do you know of another way to do what the OP asked (admittedly several years ago)? I'm looking at the same problem. $\endgroup$ – thecity2 Jul 12 '16 at 23:01
  • $\begingroup$ @thecity2 Although I have thougth about this problem for a while, I must sadly say that I do not have a solution yet. If you find one, then please let me know!!! $\endgroup$ – Fabian Werner Jul 22 '16 at 13:54
  • $\begingroup$ @FabianWerner You don't have a train and test set in this problem - your comment is irrelevant here. This is unsupervised learning, just like clustering. $\endgroup$ – Eran Marom May 14 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.