1
$\begingroup$

Is there a way to find out what sentences fall under which topic detected using Latent Dirichlet Allocation (LDA)?

Assume I have already used LDA to extract topics. Now I want to determine which sentences in my document (single document) fall under these topics (let's say top 5 or 10 topics). How do I do that?

I'm using gensim library for LDA.

Also, Would applying LDA separately on each sentence of the document yield different results (i.e. topics extracted) from if I applied LDA to the whole document?

$\endgroup$
1
$\begingroup$

Yes, it is possible to assign topics to sentences, or, more generally, to give each sentence a probability of belonging to each topic. Many LDA inference methods provide a probability of each word belonging to each topic, which you can simply aggregate by averaging to determine the probability of each sentence belonging to each topic. If you want to assign a single topic to each sentence, you can simply choose the topic with the highest probability; how you break ties is up to you.

I am not an expert in gensim, but that project appears to use variational inference for LDA. In this case, you will want the variational parameter giving a probability distribution on words belonging to topics, but I don't see by glancing through the docs/source how to attain this.

Here's the heuristic I would use: just look at the matrix relating terms to topics, and for each sentence, add up the topic contributions of each term. This ignores information about other sentences in the document, but it should be a reasonable approximation. Consult the method "get_term_topics" belonging to the LDA object to obtain this.

Is LDA on sentences equivalent to LDA on documents? The answer here is no. In deciding what topic each word of the corpus comes from, LDA inference algos borrow information from what other words are in the corpus through a parameter (denoted by $\theta$ in the original LDA paper) which gives the topic prevalence for each document. Therefore, doing LDA with each sentence being considered its own document will give a different result, since sentences won't "borrow strength" from one another. I would conjecture that you will get a somewhat similar result, but it won't be the same. Further, standard LDA inference algos have difficulty with short documents (such as tweets, which are sometimes aggregated so as to have longer docs, see e.g. this article), so you may see some degradation in the quality of the results.

$\endgroup$
  • $\begingroup$ Thankyou for your answer. I was wondering if applying LDA seperately on each sentence yield the same results as the approach you suggested since that way we would also be ignoring corelation between sentences? Would this be a feasible solution? $\endgroup$ – user233222 Jan 7 at 8:32
  • $\begingroup$ @user233222 Apologies for missing that part of your question, I will answer it in an edit to my post. $\endgroup$ – John Madden Jan 7 at 18:04
  • $\begingroup$ Thankyou so much. This has really helped a lot $\endgroup$ – user233222 Jan 8 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.