4
$\begingroup$

I was taking a machine learning course and they say the following two phrases that confuse me:

each document is a distribution on topics.

and

each topic is a distribution on words.

I was wondering if someone knew what that meant.

Here is a link to the notes:

http://people.csail.mit.edu/moitra/docs/bookex.pdf


Currently this is how I interpret it (my thoughts).

Well, we are modeling a topic as a vector $u^{(i)}$ with the relative frequencies of each word. So it just specifies how often each word appears in a specific topic. Also, each document can approximately thought of as a linear combination of these topic vectors, i.e. document $M_{j} = \sum^{r}_{i=1} w_{i}u^{(i)}$

thought I wasn't sure if that was right or how to include the concept of "distribution" to this.

$\endgroup$
  • $\begingroup$ I can find references to "distribution on words" at the top of p. 9 and at pp 19-20, but I see no reference to a "distribution on topics." Could you please provide accurate quotations from your notes? $\endgroup$ – whuber Feb 28 '15 at 21:26
  • $\begingroup$ I won't lie but probably the second quote I heard during lecture. I can provide a link to the lecture video but I don't think I remember when it was said exactly: youtube.com/… $\endgroup$ – Pinocchio Mar 4 '15 at 16:29
8
+50
$\begingroup$

Typically, in the context of Latent Dirichlet Allocation (used for Topic Modeling), we assume that the documents come from a generative process. I'll avoid math notation. Look at this figure:

generative_process

  • (1) Every topic is generated from a Dirichlet distribution of $V$ dimensions where $V$ is the size of your vocabulary.

  • (2) For every document:

    • (2.1) Generate a distribution over topics from a Dirichlet distribution of $T$ dimensions where $T$ is the number of topics in the corpus.
    • (2.2) For every word in the document:
      • (2.2.1) Choose a topic according to the distribution generated at (2.1)
      • (2.2.2) Choose a word according to the distribution corresponding to the chosen topic (generated at (1))

The rigorous mathematical explanaition is here (section 3).

So, each topic is a probability distribution over the words of the vocabulary (1) because it says the probability, in that topic, of the word "dog" to appear.

And each document has a probability distribution over topics (2.1) which says from which topics the document is more likely to draw its words. We say that a document is a mixture of topics

Note:

  • A Dirichlet distribution of three dimensions draws thinks like [0.2,0.4,0.4], [0.3,0.3,0.4], etc. which can be used as Categorical distributions. This is why it is used to generate distributions over $V$ words (topics), and distributions over $T$ topics. See left and right sides of the figure.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.