Understanding original LDA article

Question

I decided to write a different question as a follow up to a comment here about LDA :

Upgrading weight parameters to random variable in Gaussian mixtures

I am trying to read about latent dirichlet allocation here : https://web.archive.org/web/20120207011313/http://jmlr.csail.mit.edu/papers/volume3/blei03a/blei03a.pdf .

and I wanted to understand the comment on page 997 :

It is important to distinguish LDA from a simple Dirichlet-multinomial clustering model. A classical clustering model would involve a two-level model in which a Dirichlet is sampled once for a corpus, a multinomial clustering variable is selected once for each document in the corpus, and a set of words are selected for the document conditional on the cluster variable. As with many clustering models, such a model restricts a document to being associated with a single topic. LDA, on the other hand, involves three levels, and notably the topic node is sampled repeatedly within the document. Under this model, documents can be associated with multiple topics.

So the graphical model of LDA is:

, where $M$ is the number of documents and $N$ the number of words in a document (should be equal to the image in the article).

QUESTION: Is the Dirichlet-multinomial clustering model that the paragraph refers to the modification :

so that the only difference is that all words in a document are sampled from the same topic ???? My main issue is that I do not know this clustering model and to me is not simple, quoting the text, hence my doubt that I am not getting the point ... at least to me is not simpler than vanilla LDA...

Answer 1

Your question is vague. LDA models you have suggest to me you are modeling topics. Topics is distribution over words that are present in text where relevant words do have higher probability. These topics are common for all text and should be $z$.

It is important to distinguish LDA from a simple Dirichlet multinomial clustering model.

It is because LDA can do many things:

• Dimensionality reduction for document representation
• Search for similar documents in the corpus
• Document clustering
• Generation of coherent texts on particular topic

Well you can do LDA with Dirichlet multinomial clustering in which case you will be using this Dirichlet prior.

$p(\theta)=\operatorname{Dir}(\theta \mid \alpha)=\frac{1}{B(\alpha)} \prod_{k=1}^{K} \theta_{k}^{\alpha_{k}-1}$