I have an understanding into the technicals of word2vec. What I don't understand is:

Why semantically similar words should have high cosine similarity. From what I know, goodness of a particular embedding is seen in shallow tasks such as word analogy. I am unable to grasp the relationship between maximizing cosine similarity and good word embeddings.


Why semantically similar words should have high cosine similarity.

From wikipedia on distributional semantics:

The distributional hypothesis in linguistics is derived from the semantic theory of language usage, i.e. words that are used and occur in the same contexts tend to purport similar meanings.[1] The underlying idea that "a word is characterized by the company it keeps" was popularized by Firth.

Why exactly cosine similarity? Because apart from being a similarity, which is in itself useful, it is related to euclidean distance: if $$\|x\| = \|y\| = 1$$ then $$\|x-y\|^2 = 2 - 2 \langle x, y\rangle$$, because

$$\|x-y\|^2 = \langle x-y, x-y \rangle = \|x\|^2 + \|y\|^2 - 2 \langle x, y\rangle$$

To sum up: word2vec and other word embedding schemes that tend to have high cosine similarity for words that occur in similar context - that is, they translate words which are similar semantically to vectors that are similar geometrically in euclidean space (which is really useful, since many machine learning algorithms exploit such structure).


One of the main difference in cosine based similarity is the non-affect the dual 0 bits have(There is no angle at 0).
In the case of word-similarities, it helps the algorithm focus only on sentences(or phrases or documents or..) where at least one of the words exist.
This in a contrary to using the Euclidean norm(for example), where all the sentences were none of the words exist, increase their similarity(Because they don't exist together).

Imagine you want to know how similar is Cat to Dog. is the phrase:
"My milkshake brings all the boys to the yard" should support this similarity?
If you think not, then using a cosine based similarity might be a good practice.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.