0
$\begingroup$

I have a corpus and number of documents with me. I am trying to generate a similarity score between the corpus and each of the available documents using latent dirichlet allocation. The goal is to see how similar the documents are to the corpus. The document that is very similar gets a high similarity score and the one that isn't gets a low similarity score.

I was able to generate a topic model for the corpus

[(0, '0.131*"run" + 0.046*"fast" + 0.045*"woods"'), (1, '0.158*"great" + 0.036*"jump" + 0.029*"sea"'), (2, '0.058*"car" + 0.051*"quality" + 0.030*"engine"')]

and then applied this lda model on the documents to get topic probability distributions

[(0, 0.76487518533846643), (1, 0.19539064775895373), (2, 0.039734166902579866)]
[(0, 0.52023195599405925), (1, 0.41332822829405069), (2, 0.066439815711890018)]
[(0, 0.45468263525907138), (1, 0.5359866943966648)]
[(0, 0.57673377931050251), (1, 0.16880237753053462), (2, 0.25446384315896298)]
[(0, 0.31649406267968366), (1, 0.26830459132208501), (2, 0.41520134599823139)]

But I just need "one number" representing the similarity of document to the corpus and not a probability distribution. Is there any way to generate such a similarity score?

Thanks for your help!

$\endgroup$

1 Answer 1

1
$\begingroup$

LDA does not generate a single distribution for the corpus, rather it generates a distribution over topics for each document - so you should end up with $N$ distributions where $N$ is the number of documents in your corpus. Therefore, your question to "generate a similarity score between the corpus and each of the available documents" doesn't make sense.

What you can do to compare individual documents to each other using the topic distributions of each document and the metrics below

Using these metrics will enable you to see, given a query / input document, which documents in the corpus are most similar to the query document.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.