Latent Dirichlet Allocation uses as prior for topic distribution the Dirichlet prior. However this model doesn't provide a correlation between topics and for this reason it was introduced Correlated Topic Model, that use a logistic normal distribution and provide this. I was looking at the Dirichlet distribution and the covariance between variables is not null, hence correlation can be explained. Can somebody explain me why the "upgrade" has been made then ?


From the paper introducing correlated topic models (emphasis mine):

Under a Dirichlet, the components of the proportions vector are nearly independent; this leads to the strong and unrealistic modeling assumption that the presence of one topic is not correlated with the presence of another.

In essence, the Dirichlet must have some covariance because the parameters are constrained to the simplex, but the distribution has no means of expressing distributions with the same mean but difference covariance. Hence the term "nearly independent," and why the authors use the logistic normal:

It describes correlations between components of the simplicial random variable through the covariance matrix of the normal distribution. The logistic normal was originally studied in the context of analyzing observed compositional data such as the proportions of minerals in geological samples. In this work, we extend its use to a hierarchical model where it describes the latent composition of topics associated with each document.

For an intuitive example, consider a simple corpus with three topics that occur in $\big(\frac{1}{2},\frac{1}{4},\frac{1}{4}\big)$ proportions, but such that the first and last topics almost never co-occur. A logistic normal could capture this covariance structure; a Dirichlet could not.

  • $\begingroup$ Was that paper ever published in a peer reviewed journal or on a peer reviewed conference? $\endgroup$ – jknappen Feb 10 '17 at 10:58
  • 1
    $\begingroup$ Yep, NIPS. A related paper was in Annal of Applied Statistics. $\endgroup$ – Sean Easter Feb 10 '17 at 12:20
  • $\begingroup$ @jknappen glad to help! $\endgroup$ – Sean Easter Feb 10 '17 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.