What is a good positional similarity measure? Bottom line - words, which are next to each other should be ranked higher (no semantic or fuzzy matching needed). I want to use this in addition to TF/IDF-like similarities, which operate on a single word at a time.

One idea is to split query into bigrams and compute similarity of each of them as: BF * 2 / (TF1 + TF2), where BF = number of times given bigram occurs in a text, TF1/2 - number of time first/second term of a bigram occurs. Obviously this ranges [0, 1] (if repetitions not allowed).

Same could be applied to trigrams, while 4-grams are most likely not needed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.