# Topic Words Selection in Topic Modeling

I understand how generative model of topic modeling works; for each topic there is a distribution of words, and for each document there is a distribution of topics.

Question is how words are determined for each individual topic ? Does this change algorithm to algorithm ?

Edit:

Such as:

Topic-1 : internet, web, search etc.

Topic-2 : literature, novel, etc

I wonder how "internet, web, search" are chosen for topic-1. Why "internet" and "web" take place in same topic instead of separate topics? Is it because they frequently coexist in different documents?

• idk what this means "how bag of words is determined for each individual topic". please clarify. Dec 2, 2012 at 18:26
• Are you basically asking how topic models work? Dec 4, 2012 at 22:46
• I think he is asking how to algorithmically pick the descriptive words inside a particular topic to summarize the topic. Dec 5, 2012 at 0:31
• In response to your flag, metdos, I would suggest to wait some days--at least the time of the bounty--before taking any action on your question (which you can do yourself by the way).
– chl
Dec 6, 2012 at 22:44

In standard topic modeling, each topic is a discrete probability distribution over the entire vocabulary. However, after performing inference it is generally the case that only a few words have significant probability mass in each topic. For example:

Word        Probability
----------  -----------
internet    0.2
web         0.15
search      0.08
.
.
milkshake   0.000002


The above topic assigns very high probability to words related to the internet, but it also assigns some non-zero probability to every other word in the vocabulary (including, for example, milkshake).

Since most words in the vocabulary have very small probability under this topic, we can summarize the topic by only showing the highest-probability words. Such summaries might make it look like each topic is a distribution over a different subset of words, which I think is the source of your confusion.

• But how algorithm decides to increase probability of "internet" and "web" for topic-x instead of topic-y ? Dec 5, 2012 at 7:33
• You can try to look for more explanation on en.wikipedia.org/wiki/Latent_Dirichlet_allocation
– ThiS
Dec 5, 2012 at 10:00
• Ah, I see you are really just asking for an explanation of how topic modeling "works". There are many explanations with varying levels of detail already available online. I suggest you read one or two, and then come back here if you have specific questions that are hindering your understanding.
– Drew
Dec 5, 2012 at 18:22
• Drew, I up voted your answer. I wonder, what is the best way of choosing a cut-off value? stats.stackexchange.com/questions/199263/… Mar 1, 2016 at 19:30

Yes, it is because they frequently co-occur across documents. LDA is a probabilistic model that describes a process for how documents are generated, namely by randomly choosing a set of topics and then randomly choosing words from those topics. The topics themselves are hidden variables which seek to account for why groups of words tend to co-occur together within documents. So, the goal of topic modeling is to use some inference method that seeks to discover the hidden topics that best explain the co-occurrence patterns of words across documents. It is closely related to probabilistic latent semantic analysis.