Finding similar text - algorithms and evaluation

Question

I've been asked to create a program that will rank similar texts to an input text given a collection of text.

So far I've been using a tdidf representation and cosine similarity with a lot of regex-based cleaning. The vocabulary is extremely domain specific, with a lot of codes, equipment names and tables.

The major problem that I find is to prevent from retrieving similar text based on unimportant parts of the input. For example, for "I have a problem on machine ABCD module XYZ" I wouldn't like to retrieve "I have a problem on machine EDFG module WNM". But maybe I would like to retrieve "ABCD has issues in XYZ".

I've been solving this by identifying the problematic cases and adding them to a cleaning function. For example, I could remove words {"I", "have", "a", "problem", "on"} from vocabulary as I don't want them to contribute to my similarity function. This approach is obviously not very scalable.

The second problem that I face is how to figure out the performance of the model.

Answer 1

I do not know what language you are implementing your algorithm in, but below are some suggestions with accompanying Python pointers as to how you can implement them:

1.Apply out-of-the-box stop-word removers, which would get rid of the pronouns, prepositions etc. So you do not need to reinvent the wheel, below is a Python implementation example:

import gensim
tokens = [t for t in tokens if t not in gensim.parsing.preprocessing.STOPWORDS]

2.Filter your dictionary based on frequencies, below is an example. This loosely serves as regularization. You can filter out very infrequent or frequent terms as below:

dictionary = gensim.corpora.Dictionary(docs)
dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)

3.It is common to apply Part of Speech tagging and filter out terms that have unwanted tags or weight differently based on the tag. For instance for some sentiment analysis tasks, programmers over-weight adjectives and verbs over other terms. Simple example is below:

import ntlk
text = word_tokenize("And now for something completely different")
nltk.pos_tag(text)
>>[('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'),
('completely', 'RB'), ('different', 'JJ')]

Finally, having your application-specific "cleaning function" is reasonable, as long as you are utilising what is already out there and only using the cleaning function to complement these.