###LDA does not have a distance metric
The intuition behind the LDA topic model is that words belonging to a topic appear together in documents. Unlike typical clustering algorithms like K-Means, it does not assume any distance measure between topics. Instead it infers topics purely based on word counts, based on the bag-of-words representation of documents.
This can be appreciated from the Gibbs sampler described in paper by Griffiths et al.:
$$ P(z_i=j \mid \textbf{z}_{-i} , \textbf{w} ) \propto \frac{n^{(w_i)}_{-i,j}+\beta}{n^{(.)}_{-i,j}+W\beta} \times \frac{n^{(d_i)}_{-i,j}+\alpha}{n^{(d_i)}_{-i,.}+T\alpha} $$
$P(z_i=j \mid \textbf{z}_{-i} , \textbf{w} )$ refers to the probability of assigning topic $j$ to $i^{th}$ word, given all other assignments. This depends on two probabilities:
- Probability of word $w_i$ in topic $j$
- Probability of topic $j$ in document $d_i$
These probabilities can be easily computed using the following counts:
- $n^{(w_i)}_{-i,j}:$ number of times word $w_i$ was assigned to topic $j$
- $n^{(.)}_{-i,j}:$ total number of words assigned to topic $j$
- $n^{(d_i)}_{-i,j}:$ number of times topic $j$ was assigned in document $d_i$
- $n^{(d_i)}_{-i,.}:$ total number of topics assigned in document $d_i$
- $T:$ number of topics
- $W:$ number of words in vocabulary
- $\alpha, \beta:$ Dirichlet hyperparameters
Note that all counts are excluding the current assignment, denoted by the $-i$ subscript.
###Why does LDA work?
Referring to these Video Lectures, David Blei attributes it to the following:
