What is the formula for c_v coherence?

Question

I've recently been playing around with Gensim LDAModel. I use coherence to evaluate the results. Gensim offers a few coherence measures. This includes c_v and u_mass.

While there is a lot of materials describing u_mass on the web, I could not find anything interesting on c_v. There must be some significant difference sice c_v is always positive and u_mass is always negative.

I'd really like to understand the c_v measure, could you please provide me with the exact formula for calculating the c_v measure?

Answer 1

In This paper , c_v measure is explained and these three equations are used to describe it.

I hope this is helpful to you and anyone who comes back to this post

Answer 2

I didn't succeed to find an exact formula, but here is at least more precise description:

CV is based on a sliding window, a one-set segmentation of the top words and an indirect confirmation measure that uses normalized pointwise mutual information (NPMI) and the cosinus similarity.

This coherence measure retrieves cooccurrence counts for the given words using a sliding window and the window size 110. The counts are used to calculated the NPMI of every top word to every other top word, thus, resulting in a set of vectors—one for every top word. The one-set segmentation of the top words leads to the calculation of the similarity between every top word vector and the sum of all top word vectors. As similarity measure the cosinus is used. The coherence is the arithmetic mean of these similarities. (Note that this was the best coherence measure in our evalution.)

Source: http://palmetto.aksw.org/palmetto-webapp/

Answer 3

The c_v coherence measure was proposed and described in a systematic framework of coherence measures by Röder et al.

The best performing coherence measure [...] is a new combination found by systematic study of the configuration space of coherence measures. This measure (CV) combines the indirect cosine measure with the NPMI and the boolean sliding window. This combination has been overlooked so far in the literature.

I recommend to read the original paper to understand the different steps (segmentation, probability calculation, confirmation measure and aggregation) behind this measure and others, such as u_mass, c_uci or c_npmi, as well. The authors use a consistent notation to describe all coherence measures in a systematic framework. You will find the formulas in the paper.

⚠️ However, there are a couple of issues with this coherence measure. Therefore, the authors don't recommend to use it anymore.

It's not recommended to use CV because there are known issues associated with it.

Röder, M., Both, A., & Hinneburg, A. (2015). Exploring the space of topic coherence measures. WSDM 2015 - Proceedings of the 8th ACM International Conference on Web Search and Data Mining, 399–408. https://doi.org/10.1145/2684822.2685324

Answer 4

The algorithm remained vague to me until I dove into the code. See my blogpost to walk through the mathematics of cᵥ step-by-step. In words, I would describe it as follows:

In cᵥ coherence, each topic word is compared with the set of all topics. A boolean sliding window of size 110 is used to assess whether two words co-occur. Then, the confirmation measure consists of a direct- and indirect confirmation. That is, for all N most probable words per topic, a ‘word vector’ of size N is created in which each cell contains the Normalized Pointwise Mutual Information (NPMI) between that word and word i, i in{1,2,…,N}. Then, all the word vectors in a topic are aggregated into one big topic vector. The average of all the cosine similarities between each topic word and its topic vector (this is the segmentation) is used to calculate the Cᵥ score.