I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning.

I tried several approaches: using tfidf distance between sentences, doc2vec distance, the "word movers" distance. For the last two approaches I used the gensim package. I trained word2vec encoding using my corpus starting from scratch, and alternatively, starting from the GoogleNews-vectors-negative300 embedding and further training it on my corpus.

All three approaches do find sentences with similar meaning. However, they all fail in one regard. They fail to differentiate between "important" and "not important" words in the sentences.

Here is one example. The "seed" sentence is

Ensure records created by an imaging system are retained and destroyed in accordance with the approved standards

Two of the sentences with a high similarity score

Ensure physical records are stored or transferred to an approved records repository


Review the output of the imaging process to ensure they produce consistent and usage documents

While I can't argue that there is some similarity in the context of the seed sentence and the matches, the matches only captured one of two important aspects of the seed sentences, which is, the seed sentence is about storing imaging products. Each match only capture one of the aspects, it's either imaging or storage.

Basically when the tfidf distance is computed or when the word-mover algorithm matches words with close word embedding, they cannot discriminate between "important" and "not so important" words.

I'm curious if there is any research out there that attempts to address this problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.