2
$\begingroup$

I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation.

I have often read that the cosine distance, ergo the angle between vectors, is used for calculate similarity. I'll illustrate my problem with this example, which shows a simplified representation in the latent word2vec space and discriminates between farm animals and pets:

enter image description here

Here the vector for "Dog" has approximately the same direction as the vector for "Cow", thereby they would be classified as very similar in terms of cosine distance. Also, the distance between "Cat, Dog" would be way bigger than ""Cow, Horse", since they are closer to the origin.

Considering this example, how is cosine distance a valid distance measure in word2vec?

$\endgroup$

1 Answer 1

1
$\begingroup$

The cosine measure is a similarity measure, not a distance measure. The cosine similarity between two vectors $x,y$ is $$\mathrm{S}_{\cos}(x,y) = \dfrac{\langle x,y\rangle}{\lVert x \rVert\lVert y \rVert }.$$ In fact, the actual angle $\theta$ between $x$ and $y$ is equal to the similarity up to a bijection: $$\theta(x,y) = \cos^{-1}\mathrm{S}_{\cos}(x,y)$$

If you want a distance, use the normalised distance $$ \mathrm{S}_{\cos}(x,x) +\mathrm{S}_{\cos}(y,y) - 2\mathrm{S}_{\cos}(x,y)$$ Which is equal to the Euclidean distance between the normalised vectors $x/\lVert x\rVert$ and $y/\lVert y\rVert$.

This distance is equal to $ 2(1 - \mathrm{S}_{\cos}(x,y)) $ Indeed:

  • Large angle, very dissimilar
  • Small angle, very similar
$\endgroup$
3
  • $\begingroup$ In this context, what is the difference between distance measure and similarity measure? I understand that the word2vec encoding groups similar words together, so the smaller the distance the greater the similarity, or am I wrong there? $\endgroup$
    – Josef
    Commented Feb 10, 2021 at 14:30
  • $\begingroup$ A measure and distance function - metric are often confused. We can talk about "similarity" either with similarity measure or with a "similarity metric" (iif real valued function represents metric, then it is "similarity metric"). $\endgroup$ Commented Dec 1, 2021 at 6:01
  • $\begingroup$ But why use cosine similarity instead of just the Euclidean distance? (I am asking both on theoretical and practical considerations.) Doesn't the cosine similarity implicitly that there exists a meaningful zero in the vector space? $\endgroup$
    – Arthur
    Commented Jun 16, 2022 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.