6
$\begingroup$

Since a similarity/distance measure is crucial for every clustering algorithm, I wonder what this measure is for LDA. Since LDA works on text as a bag-of-word model, can someone imagine the similarity between topics (clusters) are the representative words between those clusters?

For example:

  • topic 1: [dog, cat, animal]
  • topic 2: [dog, fetch, catch]

If those are the representative words, is the measure for clustering those topics, the similarity between them in vector space?

Greetings

$\endgroup$
3
  • $\begingroup$ LDA doesn't measure distance, and doesn't do clustering. Every word belongs to every cluster with some (often tiny) probability. So your question is all but clear. $\endgroup$ Jul 19 '17 at 18:18
  • $\begingroup$ From my understanding, exactly thats the reason why LDA holds clustering characteristics. Of course it is no hard clustering like k-Means. If you feed new documents into a trained LDA model, it would give you some topic assignment-probabilities. From those, you then could cluster document similairties...right? Thanks for your comment $\endgroup$
    – Lisa
    Jul 20 '17 at 9:21
  • $\begingroup$ Although this question is based on a lack of understanding about LDA (which is what we're here for), that misunderstanding can be cleared up. The upvoted answer below is an existence proof that it can be addressed. I'm voting to leave open. $\endgroup$ Jul 20 '17 at 14:51
7
$\begingroup$

LDA does not have a distance metric

The intuition behind the LDA topic model is that words belonging to a topic appear together in documents. Unlike typical clustering algorithms like K-Means, it does not assume any distance measure between topics. Instead it infers topics purely based on word counts, based on the bag-of-words representation of documents.

This can be appreciated from the Gibbs sampler described in paper by Griffiths et al.:

$$ P(z_i=j \mid \textbf{z}_{-i} , \textbf{w} ) \propto \frac{n^{(w_i)}_{-i,j}+\beta}{n^{(.)}_{-i,j}+W\beta} \times \frac{n^{(d_i)}_{-i,j}+\alpha}{n^{(d_i)}_{-i,.}+T\alpha} $$

$P(z_i=j \mid \textbf{z}_{-i} , \textbf{w} )$ refers to the probability of assigning topic $j$ to $i^{th}$ word, given all other assignments. This depends on two probabilities:

  1. Probability of word $w_i$ in topic $j$
  2. Probability of topic $j$ in document $d_i$

These probabilities can be easily computed using the following counts:

  • $n^{(w_i)}_{-i,j}:$ number of times word $w_i$ was assigned to topic $j$
  • $n^{(.)}_{-i,j}:$ total number of words assigned to topic $j$
  • $n^{(d_i)}_{-i,j}:$ number of times topic $j$ was assigned in document $d_i$
  • $n^{(d_i)}_{-i,.}:$ total number of topics assigned in document $d_i$
  • $T:$ number of topics
  • $W:$ number of words in vocabulary
  • $\alpha, \beta:$ Dirichlet hyperparameters

Note that all counts are excluding the current assignment, denoted by the $-i$ subscript.


Why does LDA work?

Referring to these Video Lectures, David Blei attributes it to the following:

Why LDA works

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.