3
$\begingroup$

Let's say I run an LDA model with 3 topics on 5 documents.

After the model is learned (with Gibbs sampling presumably), I have topic distribution for each document, shown as the following:

enter image description here

My question is, how do I retrieve document(s) that are "most similar to document-1" ?

In clustering algorithms such as K-means, each document is assigned to one of the K classes. To retrieve doc-1's neighbor documents, I just need to find all the documents that get assigned to the same cluster as doc-1.

What procedure should I do with LDA model?

Improve this question
$\endgroup$
2
  • $\begingroup$ If you have per document "topic" proportions, you could use a measure to compare discrete distributions like Earth Mover's Distance. $\endgroup$
    – kedarps
    Mar 29, 2017 at 22:08
  • $\begingroup$ Is your question about how to calculate the difference between probability distributions? (@kedarps's answer) Or generally how to compute similarity between documents? Because as I understand it, you can use your distance metric of choice with the topic proportions. See this answer. $\endgroup$
    – Ben
    Jul 25, 2017 at 20:56

1 Answer 1

Reset to default
1
$\begingroup$

I found the metrics below to be quite useful. Use these metrics to compare document 1 ($P$ distribution on Wikipedia) to document $i$ ($Q$ distribution on Wikipedia). Repeat this for all documents (iterate through $i$ and replace $Q$ each time) to create a list of distances. Then rank the distances from smallest to biggest - the smallest one will be the most similar to doc-1

Different metrics may return different results - it is up to you which one works best / suits your needs.

Improve this answer
$\endgroup$
2
  • $\begingroup$ If I understand correctly, what you mean is to calculate the distance metrics between the distribution over topics for doc-1 and the distribution over topics for doc-i, where i=2, 3,..., numb_of_docs. $\endgroup$
    – cwl
    Oct 31, 2017 at 2:49
  • $\begingroup$ Yes that's correct. Apologies if I didn't explain it clearly. Those metrics can also be simplified depending if you have sparse matrices (some of your topic contributions are zero) or if your distributions are normalized to 1. If you are coding this in python, I would be happy to post my code in the answer to show you how you can efficiently find the documents that are most similar to a given query document. $\endgroup$
    – PyRsquared
    Oct 31, 2017 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.