3
$\begingroup$

I have an understanding into the technicals of word2vec. What I don't understand is:

Why semantically similar words should have high cosine similarity. From what I know, goodness of a particular embedding is seen in shallow tasks such as word analogy. I am unable to grasp the relationship between maximizing cosine similarity and good word embeddings.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Why semantically similar words should have high cosine similarity.

From wikipedia on distributional semantics:

The distributional hypothesis in linguistics is derived from the semantic theory of language usage, i.e. words that are used and occur in the same contexts tend to purport similar meanings.[1] The underlying idea that "a word is characterized by the company it keeps" was popularized by Firth.

Why exactly cosine similarity? Because apart from being a similarity, which is in itself useful, it is related to euclidean distance: if $$\|x\| = \|y\| = 1$$ then $$\|x-y\|^2 = 2 - 2 \langle x, y\rangle$$, because

$$\|x-y\|^2 = \langle x-y, x-y \rangle = \|x\|^2 + \|y\|^2 - 2 \langle x, y\rangle$$

To sum up: word2vec and other word embedding schemes that tend to have high cosine similarity for words that occur in similar context - that is, they translate words which are similar semantically to vectors that are similar geometrically in euclidean space (which is really useful, since many machine learning algorithms exploit such structure).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Isn't ||x||=||y||=1 a big if? Doesn't cosine similarity kin-of supposes a meaningful zero in the vecto space? And even though they are related by a not-to-difficult-to-follow transformation, if one is clearly optimising one distance (say Euclidean distance), why should one use the other (say cosine similarity) ? I could not find any explicit analysis of the most natural distance in an embedding space, and would be glad for any clarification ! [I know it is an old question, but I did not find resources online on the subject] $\endgroup$
    – Arthur
    Jun 16, 2022 at 10:59
  • $\begingroup$ This equation isn't a big if because you can always force it by normalizing the vectors. I don't understand the second part though. Cosine similarity is sort of like euclidean when you don't care about vector length. $\endgroup$
    – Jakub Bartczuk
    Jun 16, 2022 at 19:57
  • $\begingroup$ As for the "natural distance", unfortunately I don't think there is such a thing. The purpose of embedding will determine optimization problem, and for some it might be more useful to choose one distance over another. Also it will depend on the data. I can point you to a particular example if you want. $\endgroup$
    – Jakub Bartczuk
    Jun 16, 2022 at 20:04
  • $\begingroup$ Thanks for the elaboration. My personal goal is to nudge a latent vector from the embedding with some random noise, but I want to do it in a way that respects the embedding's natural metric (otherwise, I may end up over-sampling rare vectors in empty parts of the space). $\endgroup$
    – Arthur
    Jun 17, 2022 at 15:15
  • 1
    $\begingroup$ Oh I understand. The vectors are initialized randomly, so that 's a good question what corresponds to zero in embedding space. If you want theoretical background I recommend papers that link word embeddings to matrix factorization. I think Levy or Goldberg had papers on that (there was some result that word2vec implicitly performs to factorizing pointwise mutual information matrix) $\endgroup$
    – Jakub Bartczuk
    Jun 18, 2022 at 11:32
-1
$\begingroup$

One of the main difference in cosine based similarity is the non-affect the dual 0 bits have(There is no angle at 0).
In the case of word-similarities, it helps the algorithm focus only on sentences(or phrases or documents or..) where at least one of the words exist.
This in a contrary to using the Euclidean norm(for example), where all the sentences were none of the words exist, increase their similarity(Because they don't exist together).

Imagine you want to know how similar is Cat to Dog. is the phrase:
"My milkshake brings all the boys to the yard" should support this similarity?
If you think not, then using a cosine based similarity might be a good practice.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.