3
$\begingroup$

I have a training data set on which I use Latent Dirichlet Allocation(LDA) to generate topics. I would like to use this model on other documents and see how similar they are to the training data. I would like to know how to go about doing this?

I was able to generate topics on the training data set. Gensim gave me a topic distribution. But how do I apply this topic distribution which is the LDA model on other documents to calculate similarity?

$\endgroup$
2
$\begingroup$

(Pseudo-code) Computing similarity between two documents (doc1, doc2) using existing LDA model:

  1. lda_vec1, lda_vec2 = lda(doc1), lda(doc2)
  2. score <- similarity(lda_vec1, lda_vec2)

In the first step, you simply apply your LDA model on the two input documents, getting back a vector for each document. The vector represents the topic distribution for the document.

The second step is to apply a similarity measure of your choice to compare the two vectors. You should experiment with different types of similarity measures to see which one works best in your case. Some good options to consider for distance metrics are cosine distance and Hellinger distance. Note that the underlying assumption here is that we consider two documents to be similar if their presumed topics are similar.

Example using Cosine similarity:

similarity = gensim.matutils.cossim(lda_vec1, lda_vec2)
$\endgroup$
0
$\begingroup$

Any similarity measure based on LDA will have some features that are counter-intuitive to the idea of similarity:

  • They depend on the underlying topic model. Different runs of LDA on the same set of documents with different random seeds create different topic models
  • Even the topic composition of a given document in a given topic model may have some randomness in it.

When you are aware of this, the proposal of @galoosh33 gives the technical details of calculation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.