1
$\begingroup$

This question is particularly in the context of the word embedding algorithm Word2Vec.

I've noticed that many examples that are given in the original paper and other blog posts say things like vec(king) - vec(man) + vec(woman) = vec(queen) or vec(Madrid) - vec(Spain) + vec(France), but how is it determined when to use addition and when to use subtraction?

Intuitively, one can deduce that subtracting the country name from the capital and adding another country name would provide information on the new country's capital, but I was wondering if there was a formal explanation on how exactly this process goes.

$\endgroup$
3
$\begingroup$

There is not really a formal explanation, there is only intuition, like you have, backed up by actual computations that confirm it.

It just turns out that if you create a word embedding, that these computations work.

So, hypothetically, if 'Madrid' is mapped to [2, 3, 4], and 'Spain' is embedded as the vector [1, 1, 1], then the difference between those, [1, 2, 3], is very close to the difference between the embeddings of 'France' and 'Paris'. So you could say that the direction of [1, 2, 3], in the vector space, corresponds to the 'is the capital of' relation.

More realistically, these vectors are in 300d, not 3d as in the example above.

$\endgroup$
  • 1
    $\begingroup$ Thank you for the answer. It is a bit of a disappointment that there is no formal definition. That makes it even more amazing that the authors were able to come up with that idea. :) $\endgroup$ – Seankala Jul 9 '19 at 7:33
  • $\begingroup$ Yes, it's quite an amazing result. Imagine hoping to find such a relation and then actually being able to publish it.. $\endgroup$ – Gijs Jul 9 '19 at 11:13
  • 1
    $\begingroup$ +1. Two other interesting tidbits: 1) you’re training a model to predict and it’s not very good, but it’s coefficients are the near-magical embedding, and 2) it’s based entirely on context and can lead to “synonyms” that aren’t really synonyms but rather are used in the same context. $\endgroup$ – Wayne Jul 9 '19 at 14:11
  • $\begingroup$ See mc.ai/arithmetic-properties-of-word-embeddings $\endgroup$ – kjetil b halvorsen Aug 19 '19 at 7:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.