I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation.

I have often read that the cosine distance, ergo the angle between vectors, is used for calculate similarity. I'll illustrate my problem with this example, which shows a simplified representation in the latent word2vec space and discriminates between farm animals and pets:

enter image description here

Here the vector for "Dog" has approximately the same direction as the vector for "Cow", thereby they would be classified as very similar in terms of cosine distance. Also, the distance between "Cat, Dog" would be way bigger than ""Cow, Horse", since they are closer to the origin.

Considering this example, how is cosine distance a valid distance measure in word2vec?


The cosine measure is a similarity measure, not a distance measure. The cosine similarity between two vectors $x,y$ is $$\mathrm{S}_{\cos}(x,y) = \dfrac{\langle x,y\rangle}{\lVert x \rVert\lVert y \rVert }.$$ In fact, the actual angle $\theta$ between $x$ and $y$ is equal to the similarity up to a bijection: $$\theta(x,y) = \cos^{-1}\mathrm{S}_{\cos}(x,y)$$

If you want a distance, use the normalised distance $$ \mathrm{S}_{\cos}(x,x) +\mathrm{S}_{\cos}(y,y) - 2\mathrm{S}_{\cos}(x,y)$$ Which is equal to the Euclidean distance between the normalised vectors $x/\lVert x\rVert$ and $y/\lVert y\rVert$.

This distance is equal to $ 2(1 - \mathrm{S}_{\cos}(x,y)) $ Indeed:

  • Large angle, very dissimilar
  • Small angle, very similar
  • $\begingroup$ In this context, what is the difference between distance measure and similarity measure? I understand that the word2vec encoding groups similar words together, so the smaller the distance the greater the similarity, or am I wrong there? $\endgroup$
    – Josef
    Feb 10 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.