2
$\begingroup$

Let's say I run an LDA model with 3 topics on 5 documents.

After the model is learned (with Gibbs sampling presumably), I have topic distribution for each document, shown as the following:

enter image description here

My question is, how do I retrieve document(s) that are "most similar to document-1" ?

In clustering algorithms such as K-means, each document is assigned to one of the K classes. To retrieve doc-1's neighbor documents, I just need to find all the documents that get assigned to the same cluster as doc-1.

What procedure should I do with LDA model?

$\endgroup$
  • $\begingroup$ If you have per document "topic" proportions, you could use a measure to compare discrete distributions like Earth Mover's Distance. $\endgroup$ – kedarps Mar 29 '17 at 22:08
  • $\begingroup$ Is your question about how to calculate the difference between probability distributions? (@kedarps's answer) Or generally how to compute similarity between documents? Because as I understand it, you can use your distance metric of choice with the topic proportions. See this answer. $\endgroup$ – Ben Jul 25 '17 at 20:56
1
$\begingroup$

I found the metrics below to be quite useful. Use these metrics to compare document 1 ($P$ distribution on Wikipedia) to document $i$ ($Q$ distribution on Wikipedia). Repeat this for all documents (iterate through $i$ and replace $Q$ each time) to create a list of distances. Then rank the distances from smallest to biggest - the smallest one will be the most similar to doc-1

Different metrics may return different results - it is up to you which one works best / suits your needs.

$\endgroup$
  • $\begingroup$ If I understand correctly, what you mean is to calculate the distance metrics between the distribution over topics for doc-1 and the distribution over topics for doc-i, where i=2, 3,..., numb_of_docs. $\endgroup$ – cwl Oct 31 '17 at 2:49
  • $\begingroup$ Yes that's correct. Apologies if I didn't explain it clearly. Those metrics can also be simplified depending if you have sparse matrices (some of your topic contributions are zero) or if your distributions are normalized to 1. If you are coding this in python, I would be happy to post my code in the answer to show you how you can efficiently find the documents that are most similar to a given query document. $\endgroup$ – PyRsquared Oct 31 '17 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.