0
$\begingroup$

I decided to write a different question as a follow up to a comment here about LDA :

Upgrading weight parameters to random variable in Gaussian mixtures

I am trying to read about latent dirichlet allocation here : https://web.archive.org/web/20120207011313/http://jmlr.csail.mit.edu/papers/volume3/blei03a/blei03a.pdf .

and I wanted to understand the comment on page 997 :

It is important to distinguish LDA from a simple Dirichlet-multinomial clustering model. A classical clustering model would involve a two-level model in which a Dirichlet is sampled once for a corpus, a multinomial clustering variable is selected once for each document in the corpus, and a set of words are selected for the document conditional on the cluster variable. As with many clustering models, such a model restricts a document to being associated with a single topic. LDA, on the other hand, involves three levels, and notably the topic node is sampled repeatedly within the document. Under this model, documents can be associated with multiple topics.

So the graphical model of LDA is:

enter image description here

, where $M$ is the number of documents and $N$ the number of words in a document (should be equal to the image in the article).

QUESTION: Is the Dirichlet-multinomial clustering model that the paragraph refers to the modification :

enter image description here

so that the only difference is that all words in a document are sampled from the same topic ???? My main issue is that I do not know this clustering model and to me is not simple, quoting the text, hence my doubt that I am not getting the point ... at least to me is not simpler than vanilla LDA...

$\endgroup$
0
$\begingroup$

Your question is vague. LDA models you have suggest to me you are modeling topics. Topics is distribution over words that are present in text where relevant words do have higher probability. These topics are common for all text and should be $z$.

It is important to distinguish LDA from a simple Dirichlet multinomial clustering model.

It is because LDA can do many things:

  • Dimensionality reduction for document representation
  • Search for similar documents in the corpus
  • Document clustering
  • Generation of coherent texts on particular topic

Well you can do LDA with Dirichlet multinomial clustering in which case you will be using this Dirichlet prior.

$p(\theta)=\operatorname{Dir}(\theta \mid \alpha)=\frac{1}{B(\alpha)} \prod_{k=1}^{K} \theta_{k}^{\alpha_{k}-1}$

$\endgroup$
4
  • $\begingroup$ Thanks for your answer. What is the graphical model that you are hinting at the end ? Is it the second one I drew ? Is it different from the excerpt of the article I am reporting ? $\endgroup$ – Thomas Mar 15 at 15:39
  • $\begingroup$ Yes I confirm z are the topics :) . Do you have examples of applied LDA that goes behond / is different than the article I linked ? $\endgroup$ – Thomas Mar 15 at 15:40
  • $\begingroup$ You should not pay that many attention since to me the first image should also fit to Dirichlet multinomial clustering. $\endgroup$ – Good Luck Mar 15 at 15:43
  • $\begingroup$ This is so called plate notation where N represents to repeat N times, and how can you possible learn about the topis if you don't take them in account. PS, I haven't checked your links just I am providing the common sense feedback. $\endgroup$ – Good Luck Mar 15 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.