# Finding similar text - algorithms and evaluation

I've been asked to create a program that will rank similar texts to an input text given a collection of text.

So far I've been using a tdidf representation and cosine similarity with a lot of regex-based cleaning. The vocabulary is extremely domain specific, with a lot of codes, equipment names and tables.

The major problem that I find is to prevent from retrieving similar text based on unimportant parts of the input. For example, for "I have a problem on machine ABCD module XYZ" I wouldn't like to retrieve "I have a problem on machine EDFG module WNM". But maybe I would like to retrieve "ABCD has issues in XYZ".

I've been solving this by identifying the problematic cases and adding them to a cleaning function. For example, I could remove words {"I", "have", "a", "problem", "on"} from vocabulary as I don't want them to contribute to my similarity function. This approach is obviously not very scalable.

The second problem that I face is how to figure out the performance of the model.

• I think you need to provide more on: I've been solving this by identifying the problems and adding cases to a cleaning function for anyone to suggest improvements on your approach – Zhubarb Jan 28 at 10:38
• Thanks. I edited the question, so now it's hopefully more understandable. What I tried to say is that I try to remove parts of the text which contains tokens that I don't want to contribute to the similarity. – jcp Jan 28 at 12:40

I do not know what language you are implementing your algorithm in, but below are some suggestions with accompanying Python pointers as to how you can implement them:

1.Apply out-of-the-box stop-word removers, which would get rid of the pronouns, prepositions etc. So you do not need to reinvent the wheel, below is a Python implementation example:

import gensim
tokens = [t for t in tokens if t not in gensim.parsing.preprocessing.STOPWORDS]


2.Filter your dictionary based on frequencies, below is an example. This loosely serves as regularization. You can filter out very infrequent or frequent terms as below:

dictionary = gensim.corpora.Dictionary(docs)
dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)


3.It is common to apply Part of Speech tagging and filter out terms that have unwanted tags or weight differently based on the tag. For instance for some sentiment analysis tasks, programmers over-weight adjectives and verbs over other terms. Simple example is below:

import ntlk
text = word_tokenize("And now for something completely different")
nltk.pos_tag(text)
>>[('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'),
('completely', 'RB'), ('different', 'JJ')]


Finally, having your application-specific "cleaning function" is reasonable, as long as you are utilising what is already out there and only using the cleaning function to complement these.

• Thanks! In fact I'm already doing points 1 and 2. I didn't try POS tagging because the data has a lot of very specific tokens, but I'll try. The problem with the custom cleaning is that I first have to identify the necessary cases and I could keep creating specific cleaning steps for specific cases for weeks. I also lack of a way of evaluating the performance. – jcp Jan 29 at 8:41