I am trying to figure out the details of LDA and have been stuck for a while now. While reading the paper by Blei, I came across this -
Latent Dirichlet allocation (LDA) is a generative probabilistic model of a corpus. The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized by a distribution over words. LDA assumes the following generative process for each document w in a corpus D:
- Choose N ∼ Poisson(ξ)
- Choose θ ∼ Dir(α)
For each of the N words wn:
Choose a topic zn ∼ Multinomial(θ).
Choose a word wn from p(wn |zn,β), a multinomial probability conditioned on the topic zn.
When they mention
Choose θ ∼ Dir(α), they mean that the probabilities come from a Dirichlet distribution and in case of an observation (or any number of obs), we update the α and the probabilities change accordingly. And when they say
Choose N ∼ Poisson(ξ), they mean the number of words in each document can be sampled from a Poisson distribution, that is if I have the number of words in each document they will follow a Poisson distribution. So this part I get (or correct me if I have gotten the details wrong).
Gaps in my understanding:
I want to understand what the authors mean when they say choose a topic zn ~ Multinomial(θ). From my understanding of the multinomial distribution parameterized by θ, you get the probability of a particular distribution of events (ex: picking a ball from a bag with replacement n times). So how do we choose a topic for each word using a multinomial distribution?
From what I understand each document is randomly assigned topic probabilities and each topic is assigned word probabilities and is updated later on. My question is how are these probabilities updated?