23
$\begingroup$

Not Sure if this is the right stack site, but here goes.

How does the .similiarity method work?

Wow spaCy is great! Its tfidf model could be easier, but w2v with only one line of code?!

In his 10 line tutorial on spaCy andrazhribernik show's us the .similarity method that can be run on tokens, sents, word chunks, and docs.

After nlp = spacy.load('en') and doc = nlp(raw_text) we can do .similarity queries between tokens and chunks. However, what is being calculated behind the scenes in this .similarity method?

SpaCy already has the incredibly simple .vector, which computes the w2v vector as trained from the GloVe model (how cool would a .tfidf or .fasttext method be?).

Is the model simply computing the cosine similarity between these two w2v, .vector, vectors or comparing some other matrix? The specifics aren't clear in the documentation; any help appreciated!

$\endgroup$
1
  • 1
    $\begingroup$ "how cool would a .tfidf or .fasttext method be?" the docs provide an example of replacing the GloVe vectors with FastText. It's maybe not exactly the same as having them together. Github $\endgroup$
    – Carl G
    Apr 15, 2018 at 21:46

2 Answers 2

19
$\begingroup$

Found the answer, in short, it's yes:

Link to Source Code

return numpy.dot(self.vector, other.vector) / (self.vector_norm * other.vector_norm)

This looks like it's the formula for computing cosine similarity and the vectors seem to be created with SpaCy's .vector which the documentation says is trained from GloVe's w2v model.

$\endgroup$
6
$\begingroup$

By default it's cosine similarity, with vectors averaged over the document for missing words.

You can also customize this, by setting a hook to doc.user_hooks['similarity']. This pipeline component wraps similarity functions, making it easy to customise the similarity:

https://github.com/explosion/spaCy/blob/develop/spacy/pipeline.pyx#L50

$\endgroup$
1
  • $\begingroup$ Tecnically, you appear to have linked to the SentenceSegmenter strategy. $\endgroup$
    – Carl G
    Apr 15, 2018 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.