Topic Models: Latent Dirichlet Allocations

Question

I am trying to figure out the details of LDA and have been stuck for a while now. While reading the paper by Blei, I came across this -

Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized by a distribution over words. LDA assumes the following generative process for each document w in a corpus D:

1. Choose N ∼ Poisson(ξ)
2. Choose θ ∼ Dir(α)

3. For each of the N words w_n:

   • Choose a topic z_n ∼ Multinomial(θ).

   • Choose a word w_n from p(w_n |z_n,β), a multinomial probability conditioned on the topic z_n.

When they mention Choose θ ∼ Dir(α), they mean that the probabilities come from a Dirichlet distribution and in case of an observation (or any number of obs), we update the α and the probabilities change accordingly. And when they say Choose N ∼ Poisson(ξ), they mean the number of words in each document can be sampled from a Poisson distribution, that is if I have the number of words in each document they will follow a Poisson distribution. So this part I get (or correct me if I have gotten the details wrong).

Gaps in my understanding:

1. I want to understand what the authors mean when they say choose a topic z_n ~ Multinomial(θ). From my understanding of the multinomial distribution parameterized by θ, you get the probability of a particular distribution of events (ex: picking a ball from a bag with replacement n times). So how do we choose a topic for each word using a multinomial distribution?

2. From what I understand each document is randomly assigned topic probabilities and each topic is assigned word probabilities and is updated later on. My question is how are these probabilities updated?

Answer 1

To the first question, your confusion seems due to the use of the term "multinomial" in the paper. Somewhat confusingly, "multinomial" is used for the multinomial and categorical distributions. (The two are analogous to binomial and Bernoulli.)

So what the authors mean is that each word in the document is assumed to have a single topic sampled according to $\theta_d$. (We then sample a word from the topic's corresponding distribution.) The document has a mean distribution over possible topics; each word is assumed assigned to a single, unknown topic.

As written, your second question is rather broad: Section 5 of the paper covers inference in depth.

Answer 2

I have been trying to implement a LDA program in python. So, far I have had little luck. I have found this youtube video rather informative and easy to understand.

I am a programmer and I do not really understand weird mathematical notations, but here is what I collected.

1. ndk = number of times in document, topic t was observed
2. vkwdn = number of times in topic k, wdn was observed
3. Assign topics randomly to each word in each document
4. Count the number of times a word was used in a topic from all docs
5. Count the number of times a document has a word of a particular topic
6. Normalize the vectors in 4 and 5 for each topic
7. Multiply M1[document][topic] with M2[topic][word] for each topic
8. You should get a probability vector, having the probability for each topic
9. Sample randomly using the vector as weights
10. Assign it to the topic of that word in that document
11. Go to step 4 unless everything has converged