1
$\begingroup$

I am using the tidytext, quanteda, and tm packages in R to analyse my corpus of 220 documents. Using the topicmodels package I have extracted key topics using LDA.

I now have a tidy dataframe that has a observations for document id, topic no, and probability (gamma) of the topic belonging to that particular document.

My goal is to use this information to compare document similarity based on topic probabilities. However, I am not sure how to do this.

This post kind of explains what I need to do:

Score the documents based on similarity to corpus using latent Dirichlet allocation

However, I am not familiar with the python libraries for doing this, preferring to stick with my R environment. Neither am I expert in advanced statistical methods. Hellinger distance appears the way to go.

$\endgroup$
1
  • $\begingroup$ Questions that are only about software (e.g. error messages, code or packages, etc.) are generally off topic here. If you have a substantive machine learning or statistical question, please edit to clarify. $\endgroup$ Sep 7, 2019 at 2:00

1 Answer 1

1
$\begingroup$

LDA, intuitively, projects the documents to a lower dimensional space, the topic space. In particular, gamma, is the projection of each document into this space.

You can then apply clustering in this space. Because the projection is a probability distribution, ideally you would use a distance metric designed for probability distributions. Hellinger distance is one of those measures.

For the code, you can use the library parallelDist in R.

If Matrix_Topics is a data frame with one row having the probability (gamma) of the topic belonging to that particular document, then the syntax would be:

distance_matrix <- parDist(Matrix_Topics, method="hellinger")
$\endgroup$
3
  • $\begingroup$ Although implementation is often mixed with substantive content in questions, we are supposed to be a site for providing information about statistics, machine learning, etc., not code. It can be good to provide code as well, but please elaborate your substantive answer in text for people who don't read this language well enough to recognize & extract the answer from the code. $\endgroup$ Sep 7, 2019 at 2:01
  • 1
    $\begingroup$ Edited, I hope it is more helpful this way :) $\endgroup$ Sep 9, 2019 at 0:06
  • $\begingroup$ Thanks, @alejandroll10. Welcome to the site. $\endgroup$ Sep 9, 2019 at 3:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.