# Topic Models: Latent Dirichlet Allocations

I am trying to figure out the details of LDA and have been stuck for a while now. While reading the paper by Blei, I came across this -

Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized by a distribution over words. LDA assumes the following generative process for each document w in a corpus D:

1. Choose N ∼ Poisson(ξ)
2. Choose θ ∼ Dir(α)
3. For each of the N words wn:

• Choose a topic zn ∼ Multinomial(θ).

• Choose a word wn from p(wn |zn,β), a multinomial probability conditioned on the topic zn.

When they mention Choose θ ∼ Dir(α), they mean that the probabilities come from a Dirichlet distribution and in case of an observation (or any number of obs), we update the α and the probabilities change accordingly. And when they say Choose N ∼ Poisson(ξ), they mean the number of words in each document can be sampled from a Poisson distribution, that is if I have the number of words in each document they will follow a Poisson distribution. So this part I get (or correct me if I have gotten the details wrong).

Gaps in my understanding:

1. I want to understand what the authors mean when they say choose a topic zn ~ Multinomial(θ). From my understanding of the multinomial distribution parameterized by θ, you get the probability of a particular distribution of events (ex: picking a ball from a bag with replacement n times). So how do we choose a topic for each word using a multinomial distribution?

2. From what I understand each document is randomly assigned topic probabilities and each topic is assigned word probabilities and is updated later on. My question is how are these probabilities updated?

• I can't tell what confuses you about the inference referenced in your second question, but if you clarify I'd be happy to try and update my answer. Cheers! – Sean Easter Dec 20 '16 at 15:40
• @SeanEaster thanks for the answer. my confusion is around how the parameters - the topic proportions across documents, distribution of words over topics get updated. I read a little on Gibbs sampling but couldn't make much out of it. This is the part that has me confused. To give an example in gradient descent we have a loss function and learning parameter to adjust the weights. How is that taking place here? – Clock Slave Dec 20 '16 at 16:30
• Actually, that's a useful example: The variational method in the paper defines a function to minimize (KL divergence), which turns out to be equivalent to maximizing log-likelihood. ("The overall procedure can thus be viewed as coordinate ascent in L," says the paper.) Does that help? – Sean Easter Dec 20 '16 at 18:46

To the first question, your confusion seems due to the use of the term "multinomial" in the paper. Somewhat confusingly, "multinomial" is used for the multinomial and categorical distributions. (The two are analogous to binomial and Bernoulli.)

So what the authors mean is that each word in the document is assumed to have a single topic sampled according to $\theta_d$. (We then sample a word from the topic's corresponding distribution.) The document has a mean distribution over possible topics; each word is assumed assigned to a single, unknown topic.

As written, your second question is rather broad: Section 5 of the paper covers inference in depth.

I have been trying to implement a LDA program in python. So, far I have had little luck. I have found this youtube video rather informative and easy to understand.

I am a programmer and I do not really understand weird mathematical notations, but here is what I collected.

1. ndk = number of times in document, topic t was observed
2. vkwdn = number of times in topic k, wdn was observed
3. Assign topics randomly to each word in each document
4. Count the number of times a word was used in a topic from all docs
5. Count the number of times a document has a word of a particular topic
6. Normalize the vectors in 4 and 5 for each topic
7. Multiply M1[document][topic] with M2[topic][word] for each topic
8. You should get a probability vector, having the probability for each topic
9. Sample randomly using the vector as weights
10. Assign it to the topic of that word in that document
11. Go to step 4 unless everything has converged