2
$\begingroup$

I have a bunch of documents where some of the documents are a copy of other documents with their text jumbled up and some of the words replaced by their synonyms. Mentioned below is one such example of a sentence:

Article 1 (original) : I caught up with John Snow in town making purchases at Kingslanding Hardware store to repair a broken tractor. Snow has farmed soybeans his entire life, as did his father and their fathers. I asked him about his life on the farm.

Article 2 (duplicate) : I obtained John Snow which in city in purchases make rise of the hardware at Kingslanding to repair a broken motor tractor. Snow have soya broad beans complete life have been treated, such as its father and their fathers. I asked him concerning its life on the agriculture company.

Article 3 (duplicate) : I took for above with John Snow in the city that made purchases in the warehouse of the hardware of Kingslanding to repair an broken tractor. Snow has cultivated the soybeans its whole life, like its father and his parents. I asked to him about its life in the farm.

Article 4 (duplicate) : I caught up with myself compared to John Snow downtown making of the purchases to the kingslanding store of material to repair a broken tractor. Snow cultivated soya its life whole, just as his/her father and their fathers. I questioned it about his life with the farm.

I want to do a document similarity which ends up tagging all these documents in the same group. Any suggestions along with examples or tutorials will be greatly appreciated.

$\endgroup$
  • $\begingroup$ Synonym expansion in something like lucene should get the job done $\endgroup$ – Alexey Grigorev Sep 9 '16 at 7:01
2
$\begingroup$

using python, gensim and text8 http://mattmahoney.net/dc/text8.zip

from gensim.models import word2vec
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
sentences = word2vec.Text8Corpus('text8')
model = word2vec.Word2Vec(sentences, size=200)
#save model, to avoid re-training.
#model.save('text8.model')
#load it with: 
#model = word2vec.Word2Vec.load('text8.model')
t1 = "I caught up with John Snow in town making purchases at Kingslanding Hardware store to repair a broken tractor. Snow has farmed soybeans his entire life, as did his father and their fathers. I asked him about his life on the farm."
t2 = "I obtained John Snow which in city in purchases make rise of the hardware at Kingslanding to repair a broken motor tractor. Snow have soya broad beans complete life have been treated, such as its father and their fathers. I asked him concerning its life on the agriculture company."
s1 = set(t1.split()).intersection(model.vocab)
s2 = set(t2.split()).intersection(model.vocab)
print "similarity:", model.n_similarity(s1, s2)
#similarity: 0.823769880569

Here I applied Word2Vec (search for the project home page) - actually a python implementation from the package gensim

I used an English corpus - a "cleaned" version of first 10^9 bytes of wikipedia (read more at: http://mattmahoney.net/dc/textdata) to train word2vec in english :)

then i created 2 sets of words from the test statement that are also present in the vocabulary of the trained word2vec model and computed the cosine similarity between the two sets of words - see wikipedia on "cosine similarity"

$\endgroup$
  • 1
    $\begingroup$ For a good answer on this site, you should really add some expålanation of what your code is doing, so the answer at least in part can be understood by someone not reading the code (or knowing Python). $\endgroup$ – kjetil b halvorsen Oct 27 '15 at 10:22
  • $\begingroup$ Did you actually need to use word2vec model? It's pretty confusing to only use its vocabulary $\endgroup$ – Jakub Bartczuk Sep 4 '17 at 13:34
1
$\begingroup$

Use a topic model along with a thesaurus to catch the synonyms (might not even be needed, try without it first). Basically the topic model represents each document as a vector, after which you can apply your favorite clustering or similarity search algorithm. Here's some python code for you:

$\endgroup$
0
$\begingroup$

There are two general approaches that can be used.

First is Bag of Words model which basically transforms documents to counts of words. Its drawback is that it does not capture semantic similarity. Latent Semantic Analysis and Latent Dirichlet Allocation can be thought of extensions of this model. Scikit-learn contains algorithms that encode data using BoW approach (CountVectorizer, TfidfVectorizer etc), and in fact it contains an example (Clustering text documents using k-means) that seems to deal with problem that is very similar to yours.

The second one approach is to use word embeddings. These are dense representations of words that capture cooccurrence patterns. These are really good at capturing semantic similarity. They have a drawback of being harder to understand, and also to use them you either need pretrained word vectors or train the model, what can be pretty time-consuming. Gensim contains algorithms for word embeddings, and also for embeddings of whole documents (from what I've read just averaging word vectors for documents works OK, but I don't have any reference for that).

Using either approaches yields encoding of texts that can be directly used as input to clustering algorithms or similarity queries.

If you want to read on the theory: chapters 15-16 of Manning and Jurafsky's Speech and Language Processing cover this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.