Correlated Topic Models are a great advance on the original topic model - see Blei and Lafferty 2007 for more info.

My question is this - how does a Correlated Topic Model impact the overall distribution of topics across all documents when compared with a normal topic model? With a normal topic model (with a symmetric Dirichlet prior) the distribution of topics can be remarkably uniform, which is not always appropriate. As discussed by Wallach et al. having asymmetric priors is key to having better model fit and interpretability - it would be great if the Correlated Topic Model implicitly addressed this issue by creating a topic distribution which is less uniform.


In normal topic models (i.e. Latent Dirichlet Allocation), for document $d$, we draw a topic proportion $\theta_d$ from $Dir(\alpha)$, although $\theta_d$ is a vector sums to one, each element in $\theta_{d}$ is quite independent. As a result, a word's probability of being assigned to different topics are almost independent. (This is a big drawback, intuitively, if a word has a high probability of being assigned to the "baseball" topic, then we expect it should also have a high probability of being assigned to the "sports" topic, as those two topics are correlated.)

Correlated Topic Models addresses this by drawing $\theta_d$ from a multivariate Gaussian distribution, then each element of $\theta_d$ is correlated to each other by the covariance matrix. Of course, in Gaussian distribution, there is no guarantee that $\theta_d$ will sum to 1, so CTM also normalizes it.

I would not worry about symmetry in CTM, as each topic entangles with other topics in its own ways.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.