Score the documents based on similarity to corpus using latent Dirichlet allocation

Question

I have a corpus and number of documents with me. I am trying to generate a similarity score between the corpus and each of the available documents using latent dirichlet allocation. The goal is to see how similar the documents are to the corpus. The document that is very similar gets a high similarity score and the one that isn't gets a low similarity score.

I was able to generate a topic model for the corpus

[(0, '0.131*"run" + 0.046*"fast" + 0.045*"woods"'), (1, '0.158*"great" + 0.036*"jump" + 0.029*"sea"'), (2, '0.058*"car" + 0.051*"quality" + 0.030*"engine"')]

and then applied this lda model on the documents to get topic probability distributions

[(0, 0.76487518533846643), (1, 0.19539064775895373), (2, 0.039734166902579866)]
[(0, 0.52023195599405925), (1, 0.41332822829405069), (2, 0.066439815711890018)]
[(0, 0.45468263525907138), (1, 0.5359866943966648)]
[(0, 0.57673377931050251), (1, 0.16880237753053462), (2, 0.25446384315896298)]
[(0, 0.31649406267968366), (1, 0.26830459132208501), (2, 0.41520134599823139)]

But I just need "one number" representing the similarity of document to the corpus and not a probability distribution. Is there any way to generate such a similarity score?

Thanks for your help!

Answer 1

LDA does not generate a single distribution for the corpus, rather it generates a distribution over topics for each document - so you should end up with $N$ distributions where $N$ is the number of documents in your corpus. Therefore, your question to "generate a similarity score between the corpus and each of the available documents" doesn't make sense.

What you can do to compare individual documents to each other using the topic distributions of each document and the metrics below

• Jensen-Shannon Distance

• Hellinger Distance

• Bhattacharyya distance

Using these metrics will enable you to see, given a query / input document, which documents in the corpus are most similar to the query document.