Finding similar documents in a big data set

Question

I have many text documents and my goal is to find similar documents. Apparently it is a clustering type of question and Latent Dirichlet Allocation (LDA) is a good candidate to do that.

However my problem is a bit different. After doing the initial clustering (let's say with LDA), I will receive more documents and I need to find similar documents for those, too.

A naive solution is when I receive new documents, I re-run the LDA on both old and new documents and cluster them again, but it's very inefficient considering the fact that the amount of data I have is quite huge (in Giga bytes or more).

Any idea on how I should find similar documents in this scenario?

Answer 1

Firstly, which language are you using R, Python or java?

I recently learned to perform the same task in R.

In R, I am using 'topicmodels' package, using LDA function you can create a LDA model, you can check the documentation here: [https://cran.r-project.org/web/packages/topicmodels/topicmodels.pdf]

Now, Just create a new corpus consisting new documents to be analysed. To predict the new documents using the existing model, inside the LDA method, you can edit a control list in the argument which looks like this :

R> control_LDA_VEM <- list(

• estimate.alpha = TRUE, alpha = 50/k, estimate.beta = TRUE,
• verbose = 0, prefix = tempfile(), save = 0, keep = 0,
• seed = as.integer(Sys.time()), nstart = 1, best = TRUE,
• var = list(iter.max = 500, tol = 10^-6),
• em = list(iter.max = 1000, tol = 10^-4),
• initialize = "random")

as, you only need to predict the document-topic matrix while keeping the Topic-Word matrix same, for prediction purposes, just modify the control list as:

predict.lda <- LDA(new.dtm, 5, control = list(estimate.beta= FALSE, seed= 13, initialize= "model"), model = fitted.lda)

estimate.beta; is a logical argument, through which you can calculate the topic distribution per document, if you already have the term distribution per topic

seed; is just used to make the code reproduciable

initialize; taking in three things, 'random', 'seeded' and 'model', I have used model because I already have a fitted model which I want to use,

You can play with this more as per your requirement, I recommend reading the "topicmodels: An R Package for Fitting Topic Models" from "Journal of Statistical Software" which gives a more elaborate review and explanation of the above.

Hope this helped!

Answer 2

I suggest you to use Vowpal Wabbit which has facilities for online version of LDA. Once you learn a model, you can apply it on novel data.

Answer 3

Gensim has an online LDA model

"This module allows both LDA model estimation from a training corpus and inference of topic distribution on new, unseen documents. The model can also be updated with new documents for online training."

https://radimrehurek.com/gensim/models/ldamodel.html

Answer 4

This can (also) be approached no differently than any other classification problem. You have classes on which you can train a model with the word frequencies being the features, and when new data comes in you predict the classes.

If you explicitly want to include clustering in the process, do clustering on the word frequency matrix (could be as simple as k-means), and then see which centroid - or specific documents - the new documents (once they have been transformed into word frequencies as well) lie closest to.

Answer 5

I am not a fan of microsoft, but they open sourced a lightweight high scalability distributed toolkit for LDA: You can find it on github:

https://github.com/Microsoft/DMTK

P.S.: LDA is a Bayesian method. So in principle seeing new data corresponds to the likelihood term. The old posterior will become the prior. But how this works with a vocabulary update I don't know.