Duplicate detection/classification using TF/IDF and cosine similarity metrics

Question

I am working on text classification using TF/IDF and cosine similarity metrics. In computing the similarity between two records (objects) in a database, TF/IDF assigns weight vector for each record by assigning weight score to each attributes (strings) in a record. Cosine similarity is then used to measure the similarity of the two records from their weight vectors generated by TF/IDF.

I realised that TF/IDF does not give accurate weight score when typographical (spelling) errors are present in strings, this is because TF/IDF assigns accurate scores to words present in the dictionary (derived from the training set), hence wrongly spelt words are poorly scored, and results in the computation of inaccurate similarity between records.

I also realised that character-based similiarity metrics (affine gap, Jaro-winkler, levenshtien distance etc) are effective in computing the similarity between strings with typographical (spelling) errors, but are not effective when words are out of order in strings.

Hence, in order to measure the similarity between documents (records), I would like to enhance the effectiveness of TF/IDF to deal with typographical errors by using the error correcting ability of a character based similarity metric such as Levenshtein distance or affine gap or even Jaro distance.

Any suggestions will be highly appreciated.

Answer 1

A bit late but here would be my suggestion. What you want is to combine those two scores in a classifier. i.e you want to use both measures of similarity, rather than "enhancing" tfidf with the character based similarity. Then you can just use both of those scores as features to some ML classifier. The things to note here are that you need to make sure both scores are proper distances, so that you can fit a classifier over all examples. Tf-idf with cosine is always bounded by 0-1 as are the character based distances you listed, so you should be fine here. Just use tfidf-cosine as your x1 feature, and levenshtein/gap/jaro as x2.

One other thought is that I would expect a non-linear classifier to outperform a linear one. This is because the character distance doesn't really matter except in relation the the token based distance. If you think about it, if tfidf-cosine similarity is 1 (perfect bag of words overlap), it doesn't matter at all what the string similarity is, as that could be really low if the words are out of order, which you don't care about. Conversely, if the tfidf-cosine is low (low BOW overlap), the string similarity becomes important, as if this is really high it gives good evidence of a similar document that has misspellings. A typical example of this is in comparing e-commerce product titles, often you have a title that is the same as the other but has a different order, or has the same order but uses various abbreviations of words and thus has low tfidf. This is very similar to a XOR problem, which linear classifiers can't solve. In this case I would opt for a tree based method like Random Forest, or you could try svm/logistic regression but include the interaction terms explicitly.

Lucas