3
$\begingroup$

So Here is the LDA Generative Model enter image description here

The $\alpha$ and $\beta$ nodes represent the parameters for two Dirichlet distributions. The $\theta$ and $\phi$ nodes represent the parameters for two multinomial distributions. My question is about the $Z$ node (the topic assignment for the word). It represents a sampled topic from the topic distribution for a document from $\theta$. So I don't think it is actually a random variable, it is a sample. But then why is it a node in the generative model if it is not a random variable? This has me confused. Is $Z$ a random variable or not?

$\endgroup$
4
$\begingroup$

Under the assumed generative model, each topic $z_n$ indexes a distribution over words in the topic, and each $z_n$ is randomly drawn from $Multinomial(\theta)$. Per Blei et al, "The basic idea is that documents are represented as random mixtures over latent topics, where each topic is characterized as a distribution over words."

The keyword is "latent," i.e. unobserved. Under the assumed model, a random topic proportion $\theta$ is chosen for the document. For every word in the document, a random topic index $z_n$ is drawn, and then a random word is drawn "from $p(w_n|z_n, \beta)$, a multinomial probability conditioned on the topic $z_n$." (Blei et al again.)

But all of that is an assumed process under the model, and the only observations are the words in the documents. So even though topics are assumed sampled in the generative process, they're unobserved, random variables.

A great resource is this pair of lectures Blei gave on LDA. It's lengthy but approachable, and he spends a fair bit of time in the first half explaining the assumptions and distributions involved in the generative model.

$\endgroup$
  • $\begingroup$ If they are random variables, then what sort of distribution do they follow? $\endgroup$ – user1893354 Jun 7 '14 at 1:19
  • $\begingroup$ $z_n$ follows a $Multinomial(\theta)$ where $\theta$ is the (also latent) topic proportion for the document. $w_n$ is also multinomial, given $z_n$ and $\beta$. $\endgroup$ – Sean Easter Jun 7 '14 at 3:42
  • $\begingroup$ This makes sense. I guess it depends on how you look at the model. If you assume that Multinomial(theta) is known then Z is not really a rv because it is just a sample from a known distribution. But I suppose since Multinomial(theta) is unknown, you would have to think of Z as a rv as well. But this is kind of confusing too since it requires w to also be a rv even though it is observed. $\endgroup$ – user1893354 Jun 7 '14 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.