6
$\begingroup$

I am new to topic modeling and read about LDA and NMF (Non-negative Matrix Factorization). I understand the training process work. Let's say I have 100 documents and I want to train an LDA for these documents with 10 topics. However, I don't really understand how does this model assign topic to an unseen document?

I used Gensim. After training, I have an LDA trained model and a dictionary with most frequent words. Let's say, I have an unseen new document with the following text:

This is just a test text about topic modeling and LDA. 

Can someone explain step by step how a topic distribution is assigned to this new document in terms of algorithmic steps? The same goes for NMF method.

$\endgroup$
  • $\begingroup$ By the context, I understand that LDA refers to Latent Dirichlet Allocation, but please clarify this in the question. Also include the full name for Non-negative Matrix Factorization. $\endgroup$ – Daniel López Jan 29 '18 at 14:37
  • $\begingroup$ The Bayes decision rule of assigning topics to new documents depends on the loss function. $\endgroup$ – Łukasz Grad Jan 29 '18 at 14:49
  • $\begingroup$ LDA does not assign topics to documents, it assigns topics to words and topic-distributions to documents. $\endgroup$ – guy Jan 29 '18 at 15:23
  • $\begingroup$ @guy I should have explicitly specified that. I meant topic distribution. $\endgroup$ – nickg Jan 29 '18 at 15:25
  • $\begingroup$ The topic distribution represented as a point on the $n_{topic}$-dimensional simplex, and is inferred by looking at the posterior under a Dirichlet prior. If we were to use, say, a Gibbs sampler, the topic distribution would be updated across iterations by sampling from the associated full conditional, which by conjugacy is another Dirichlet. $\endgroup$ – guy Jan 29 '18 at 17:23
1
$\begingroup$

What you should actually do is run inference (training) on the new set of documents (the old ones and the new ones together). A short-cut that estimates this well is applying Gibbs sampling only to the new documents while using the data obtained during training unchanged, as described by @SheldonCooper in Topic prediction using latent Dirichlet allocation.

$\endgroup$
0
$\begingroup$

As already mentioned in the comments, topics are assigned to individual words in a document. The Gibbs Sampler for latent Dirichlet allocation may give some insight into this. During training, the goal is to infer topics and per-document topic proportions by repeatedly sampling topic assignments,

$$ P(z_i=j \mid \textbf{z}_{-i} , \textbf{w} ) \propto \frac{n^{(w_i)}_{-i,j}+\beta}{n^{(.)}_{-i,j}+W\beta} \times \frac{n^{(d_i)}_{-i,j}+\alpha}{n^{(d_i)}_{-i,.}+T\alpha} $$

$P(z_i=j \mid \textbf{z}_{-i} , \textbf{w} )$ refers to the probability of assigning topic $j$ to $i^{th}$ word, given all other assignments.

After convergence, the following can be estimated easily,

$$ \hat{\phi}_j^{(w)} = \frac{n_j^{(w)}+\beta}{n_j^{(.)}+W\beta} $$

$$ \hat{\theta}_j^{(d)} = \frac{n_j^{(d)}+\alpha}{n_.^{(d)}+T\alpha} $$

where, $\hat{\phi}_j^{(w)}$ is the probability of $w^{th}$ word in the $j^{th}$ topic and $\hat{\theta}_j^{(d)}$ is the probability of the $j^{th}$ topic in the $d^{th}$ document.


At testing time, the (un-normalized) posterior probability of assigning a word to topic $j$ can be easily estimated as follows,

$$ \begin{align} P(z=j \mid w, \boldsymbol{\phi}, \boldsymbol{\theta}) &\propto P(w \mid z=j, \hat{\boldsymbol{\phi}}_j) \times P(z=j \mid \hat{\boldsymbol{\theta}}^{(d)})\\ &\propto \hat{\phi}_j^{(w)} \hat{\theta}_j^{(d)} \end{align} $$

This way one can find the probability of assigning each topic to a word, and select the topic with maximum probability i.e. find the maximum a posteriori estimate.

$\endgroup$
  • 1
    $\begingroup$ The problem with this answer is that the question referred to new documents, and in your posterior calculation you use the topic distribution for a specific document d, which you do not have any estimate on when d is a new document. $\endgroup$ – Eran Marom Feb 20 '18 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.