Summary statitics to describe topic x term distribution in NLP

Question

I created a topic model which outputted 11 topics out of 437 terms on ~60000 small documents.

I wanted to show how good each topic is. But I don't know what "good" means in this case.

Here's the distribution of the relative scores of each term for its topic. (terms are in the x-axis, ordered by relevance for the topic).

It's possible to notice that some topics are well represented by a short number of highly associated terms like V1, V7, V9 and V11, while other topics are not very well clarified, like V2 and V3.

Which numerical summary statistics could help me quantify these characteristics (or other I didn't think of)?

Answer 1

Traditionally, how "good" are topics have been evaluated in a few ways depending on what you are interested in. Some are more traditional and widely used, others are more task-oriented.

1. Perplexity try to answer the question: how likely is some unseen document (i.e. test set) to use words that have high probability in my topics?. If my topics well represent the analysed domain (e.g. cinema), I would find out that the most prominent words of my topics are also words frequently used within my documents.
2. Topic coherence: how consistent/coherent are concepts described by my topics? This is a widely used metric to estimate if each word within a topic is coherent with all the other topic's words (i.e. the words describe all the same concept).
3. Pointwise Mutual Information try to answer the question: do unseen documents tend to use words from the same topic, or they tend to mix frequently words from several topics?
4. Coverage: do my topics cover the majority of words in my documents? This metric measures if your topics actually cover all the themes discussed in the corpus, or they are just focused on a few of them.
5. Topic variety: how different are topics from each other?. You can use the KL-divergence, a metric measuring how different are distributions (and your topics are actually distribution over words).

For the details about each of these metrics, you can have a look at:

• http://qpleple.com/topic-coherence-to-evaluate-topic-models/
• https://rare-technologies.com/what-is-topic-coherence/
• https://www.sheffield.ac.uk/polopoly_fs/1.294431!/file/REF_4.pdf
• Wallach, Hanna M., et al. "Evaluation methods for topic models."Proceedings of the 26th Annual International Conference on Machine Learning. ACM, 2009.