Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I understand how generative model of topic modeling works; for each topic there is a distribution of words, and for each document there is a distribution of topics.

Question is how words are determined for each individual topic ? Does this change algorithm to algorithm ?


Such as:

Topic-1 : internet, web, search etc.

Topic-2 : literature, novel, etc

I wonder how "internet, web, search" are chosen for topic-1. Why "internet" and "web" take place in same topic instead of separate topics? Is it because they frequently coexist in different documents?

share|improve this question
idk what this means "how bag of words is determined for each individual topic". please clarify. – jerad Dec 2 '12 at 18:26
I edited question, thanks. – metdos Dec 3 '12 at 7:55
Are you basically asking how topic models work? – jerad Dec 4 '12 at 22:46
I think he is asking how to algorithmically pick the descriptive words inside a particular topic to summarize the topic. – rrenaud Dec 5 '12 at 0:31
In response to your flag, metdos, I would suggest to wait some days--at least the time of the bounty--before taking any action on your question (which you can do yourself by the way). – chl Dec 6 '12 at 22:44

In standard topic modeling, each topic is a discrete probability distribution over the entire vocabulary. However, after performing inference it is generally the case that only a few words have significant probability mass in each topic. For example:

Word        Probability
----------  -----------
internet    0.2
web         0.15
search      0.08
milkshake   0.000002

The above topic assigns very high probability to words related to the internet, but it also assigns some non-zero probability to every other word in the vocabulary (including, for example, milkshake).

Since most words in the vocabulary have very small probability under this topic, we can summarize the topic by only showing the highest-probability words. Such summaries might make it look like each topic is a distribution over a different subset of words, which I think is the source of your confusion.

share|improve this answer
But how algorithm decides to increase probability of "internet" and "web" for topic-x instead of topic-y ? – metdos Dec 5 '12 at 7:33
You can try to look for more explanation on – ThiS Dec 5 '12 at 10:00
Ah, I see you are really just asking for an explanation of how topic modeling "works". There are many explanations with varying levels of detail already available online. I suggest you read one or two, and then come back here if you have specific questions that are hindering your understanding. – Drew Dec 5 '12 at 18:22
Drew, I up voted your answer. I wonder, what is the best way of choosing a cut-off value?… – Chris Mar 1 at 19:30

Yes, it is because they frequently co-occur across documents. LDA is a probabilistic model that describes a process for how documents are generated, namely by randomly choosing a set of topics and then randomly choosing words from those topics. The topics themselves are hidden variables which seek to account for why groups of words tend to co-occur together within documents. So, the goal of topic modeling is to use some inference method that seeks to discover the hidden topics that best explain the co-occurrence patterns of words across documents. It is closely related to probabilistic latent semantic analysis.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.