0
$\begingroup$

What is a good positional similarity measure? Bottom line - words, which are next to each other should be ranked higher (no semantic or fuzzy matching needed). I want to use this in addition to TF/IDF-like similarities, which operate on a single word at a time.

One idea is to split query into bigrams and compute similarity of each of them as: BF * 2 / (TF1 + TF2), where BF = number of times given bigram occurs in a text, TF1/2 - number of time first/second term of a bigram occurs. Obviously this ranges [0, 1] (if repetitions not allowed).

Same could be applied to trigrams, while 4-grams are most likely not needed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.