I'm looking for some guidance with Fasttext and NLP to help understand how the model proceed to calculate the vector of a sentence.
I'm using the fasttext method get_sentence_vector() to calculate the vector of a query sentence that I will call P1, as well as for a set of sentences (P2, P3, P4, P5, ..., Pn). Sentences can have one or more words. Then, I calculate the distance between the vector of the sentence P1 with that of each of the other sentences to finally obtain the list of the sentences closest to P1. Please note that i'm doing a preprocessing only on P1 (removal of numbers and punctuation + tokenization and lemmatization with SpaCy). The goal is to get the sentences that come closest in terms of meaning
The problem is that I do not understand the results I get for different cases:
case 1: P1 = "biofertilizers"
- distance between vectors "biofertilizers" and "chemical fertilizers" : 0.48
- distance between vectors "biofertilizers" and "bio-fertilizers" : 0.49
Here, i don't understand how vector calculated with fasttext of "biofertilizers" is closer to "chemical fertilizers" than "bio-fertilizers". Is the dash counted during the vector calculation ? Bio-fertilizers should logically be closer don't you think ?
case 2: P1 = "laptop"
- distance between vectors "laptop" and "battery chargers for laptop computers" : 0.16
- distance between vectors "laptop" and "tablet computers" : 0.27
This is not correct because "tablet computers" is closest to "laptop" in term of meaning than "battery chargers for laptop computers". Is it because the latter contain the word "laptop" that the distance is lower ?
case 3: P1 = "knives":
The distance between "knives" and "tableware, except forks, knives and spoons" is low and these two sentences are considered to be close. This should not be the case because their meanings are opposed. So I assume that Fasttext does not assimilate the negation words like 'Except' or 'not' during the vector calculation?
How does Fasttext arrive at these results when calculating distances between vectors?
I am also interested in hearing other suggestions for calculating the degree of semantic proximity between two sentences.