How does topic coherence score in LDA intuitively makes sense ?

Question

referring to: http://qpleple.com/topic-coherence-to-evaluate-topic-models/

In order to decide the optimum number of topics to be extracted using LDA, topic coherence score is always used to measure how well the topics are extracted:

$CoherenceScore = \sum_{i<j} score(w_i, w_j)$

where $w_i, w_j$ are the top words of the topic

---

There are two types of topic coherence scores:

1. Extrinsic UCI measure:

$SCORE_{UCI}(w_i, w_j) = log \frac{p(w_i, w_j)}{p(w_i)P(w_j)} $

where

$p(w_i) = \frac{D_{wikipedia}(w_i)}{D_{wikipedia}}$ and $p(w_i, w_j) = \frac{D_{wikipedia}(w_i, w_j)}{D_{wikipedia}}$

---

2. Intrinsic UMass measure:

$SCORE_{UMass}(w_i, w_j) = log \frac{D(w_i, w_j)+1}{D(w_i)} $

---

The available tutorials on the web seem to just give formulations of these measures, but do not offer further explanation as to why they are formulated like that, and why such formulation makes sense.

Can someone intuitively explain why these topic coherence scores can measure how good the chosen number of topics is??

Answer 1

The coherence score is for assessing the quality of the learned topics.

For one topic, the words $i,j$ being scored in $\sum_{i<j} \text{Score}(w_i, w_j)$ have the highest probability of occurring for that topic. You need to specify how many words in the topic to consider for the overall score.

For the "UMass" measure, the numerator $D(w_i, w_j)$ is the number of documents in which words $w_i$ and $w_j$ appear together. 1 is added to this term because we are taking logs and we need to avoid taking log of 0 when the two words never appear together. The denominator is the number of documents $D(w_i)$ appears in. So the score is higher if $w_i$ and $w_j$ appear together in documents a lot relative to how often $w_i$ alone appears in documents. This makes sense as a measure of topic coherence, since if two words in a topic really belong together you would expect them to show up together a lot. The denominator is just adjusting for the document frequency of the words you are considering, so that words like "the" don't get an artificially high score.

You could use the topic coherence scores, $CS(t)$ for $t = 1, \ldots, K$, to determine the optimal number $K^*$ of topics by finding $\arg\max_K \frac{1}{K}\sum_{t=1}^K CS(t)$. That is take the average topic coherence score for various settings of $K$ and see which gives the highest average coherence.