1
$\begingroup$

This question is particularly in the context of the word embedding algorithm Word2Vec.

I've noticed that many examples that are given in the original paper and other blog posts say things like vec(king) - vec(man) + vec(woman) = vec(queen) or vec(Madrid) - vec(Spain) + vec(France), but how is it determined when to use addition and when to use subtraction?

Intuitively, one can deduce that subtracting the country name from the capital and adding another country name would provide information on the new country's capital, but I was wondering if there was a formal explanation on how exactly this process goes.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

There is not really a formal explanation, there is only intuition, like you have, backed up by actual computations that confirm it.

It just turns out that if you create a word embedding, that these computations work.

So, hypothetically, if 'Madrid' is mapped to [2, 3, 4], and 'Spain' is embedded as the vector [1, 1, 1], then the difference between those, [1, 2, 3], is very close to the difference between the embeddings of 'France' and 'Paris'. So you could say that the direction of [1, 2, 3], in the vector space, corresponds to the 'is the capital of' relation.

More realistically, these vectors are in 300d, not 3d as in the example above.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thank you for the answer. It is a bit of a disappointment that there is no formal definition. That makes it even more amazing that the authors were able to come up with that idea. :) $\endgroup$
    – Sean
    Commented Jul 9, 2019 at 7:33
  • $\begingroup$ Yes, it's quite an amazing result. Imagine hoping to find such a relation and then actually being able to publish it.. $\endgroup$
    – Gijs
    Commented Jul 9, 2019 at 11:13
  • 1
    $\begingroup$ +1. Two other interesting tidbits: 1) you’re training a model to predict and it’s not very good, but it’s coefficients are the near-magical embedding, and 2) it’s based entirely on context and can lead to “synonyms” that aren’t really synonyms but rather are used in the same context. $\endgroup$
    – Wayne
    Commented Jul 9, 2019 at 14:11
  • $\begingroup$ See mc.ai/arithmetic-properties-of-word-embeddings $\endgroup$
    – kjetil b halvorsen ♦
    Commented Aug 19, 2019 at 7:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.