I am working on my word embedding calculation algorithm and stuck with a similarity formula. I assume that this could be easily derived formally with statistics and probabilities, but I fail to do so. Could you please help?
Given
The lexem (let’s name it collocated lexem or colex) immediately followed in the text (in different places) by lexems of interest, let’s call them lex1 and lex2*.
I know:
- How many times lex1 followed the colex (Lex1FollowCount).
- How many times lex2 followed the colex (Lex2FollowCount).
- For each lex1, lex2, colex I know how many times they occur in text ( *InText).
Goal
Based on information above I want to calculate some value (nice to be normalized in [0,1] range) which reflects how similar lexems lex1 and lex2. I am looking for formal derivation from probability theory. Including, what kind of probability theory task I am trying to solve, what I am missing and how to solve it. To be a little bit more formal, two lexems are similar (similarity = 1) when all colex lexems are collocated with the same probability with both lex1 and lex2.
One thing which makes it harder. I understand that there should be at least two outcomes: similarity and confidence and the last should be based on the amount of the considered collocations, but for the time being I want to pack them into one result, which is similarity.
So, in case for a large text I have a perfect collocation (like in the example above) which happed very rarely in text (Lex1FollowCount = 3, Lex2FollowCount=5, etc.), I’d like still have low value of the similarity. So, final similarity could be high only in case of good match confirmed by many occurrences in the text.
My best guess
I am sure that the formula could be formally derived, but I failed to do so, so I use my best guess so far:
$$ P_{lex1} = \frac{ Lex1FollowCount }{ Lex1InText } $$
$$ P_{lex2} = \frac{ Lex2FollowCount }{ Lex2InText } $$
$$ P_{correlated} = P_{lex1} * P_{lex2} $$
$$ similarity = P_{correlated} * min(Lex1InText, Lex2InText ) $$
Actually, it works and gives “good enough” results, but still could be improved if better formula used.
Problems with my formula and what needs to be taken into account
This formula hard to be normalized.
In real text two kinds of items don’t help much: (a) rare items and this is handled by the multiplication in the last formula; rare items have less result; (b) very frequent items (like “and”, “or”, etc.) which have a lot of meaning, but don’t help in words similarity calculations in most of languages. So, it would be nice if the “weight” of the most frequent items will be low, as well.
I’ve tried dozens of formulas inspired by algorithms mentioned in the Quantify the similarity of bags of words and similar, but it seems that my task in much simpler and can be finally formalized. If not, please, help me to see the real problem in my way.
