Given the graphical model of LDA (latent Dirichlet model) the graphical model We have the factorization of the joint distribution

$$P = \prod_{d = 1}^{D}P(\theta_{d}|\alpha)\prod_{n = 1}^{N}P(z_{d, n}|\theta_d)P(w_{d, n}|z_{d, n}, \beta_{1:K})\prod_{k = 1}^{K}P(\beta_{k}|\eta)$$

in which $\theta_{d}$ defines the multinomial distribution over topics for a document $d$, $z_{n, d}$ is the topic of "slot $n$" in document $d$, $w_{n, d}$ is the chosen word from topic $z_{n, d}$ for the slot $n$.

My question is:

Why we need $\prod_{k = 1}^{K}P(\beta_{k})$?

I think given only the left part of the graphical model, the probability of a given document (which is given by the bag-of-words representation) can readily be calculated.

To be specific, why is that we have $P(w_{n,d}|z_{n,d}, \beta_{1:K})$ instead of $P(w_{n,d}|z_{n,d})$?


$z_{d,n}$ is the topic assignment for word $w_{d,n}$. Knowing the topic alone is not enough information to assign a probability to $w_{d,n}$ — we also need to know how the topic is defined. In LDA, topics are defined by the categorical probability distribution they place on the set of all words in the dictionary. For the $k$th topic, this probability distribution is given by $\beta_k$, which is a vector whose length is equal to the number of words in the dictionary.

In other words, $$ P(w_{d,n}|z_{d,n}, \beta_{1:K}) = \beta_{z_{d,n}, w_{d,n}}, $$ which clearly depends on both $z_{d,n}$ and $\beta_{1:K}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.