0
$\begingroup$

Observation [1]

Word2vec is pretty good at comparing "subjects", e.g.:

$$\langle dog, pup \rangle \simeq 0.81$$

$$\langle pants, trousers \rangle \simeq 0.75$$

But it appears to be not great at encoding "relationships" in a way that preserves semantics, e.g.:

$$\langle spouse, wife \rangle \simeq 0.52$$

$$\langle employer, boss \rangle \simeq 0.27$$

$$\langle own, posess \rangle \simeq 0.26 $$

Questions

  1. Is there any obvious intuition for why this is the case?
  2. Is there any well-known way to encoding these relationship-type words in a way that the cosine distance would actually successfully measure semantic similarity?

[1] I computed these examples with the webapp here

$\endgroup$
1
$\begingroup$
  1. Is there any obvious intuition for why this is the case?

Most likely these words appear in different contexts.

  1. Is there any well-known way to encoding these relationship-type words in a way that the cosine distance would actually successfully measure semantic similarity?

You could try to operate on the embedding space, as in Piotr Migdał's blog post (Differences and projections part).

Other way to see what is in the blog post is that if you want embeddings of related words to be similar, you need to take a quotient of the word embedding space by the vector/subspace representing the relation (for example king - queen represents gender).

$\endgroup$
3
  • $\begingroup$ Would love to have some elaboration on the quotient thing. Let $V$ be the total embedding space. I'm reading your example of quotient by the relation as something like this. Let $G \equiv span(\{male, female\}) \subset V$, I consider the quotient defined by the exact sequence: $$0 \to G \hookrightarrow V \xrightarrow{\pi} V/G \to 0$$ And the idea is that we'd have $\pi(king) = \pi(queen) \in V/G$. Question: Is it actually true that the subspace $G$ contains only gender-related words? For example one might expect the word $eunuch$ to lay in that space. $\endgroup$
    – zzz
    Nov 5 '17 at 20:53
  • $\begingroup$ The idea is to quotient by $span({male - female})$ or something like that. If you do this then $\pi(male) = \pi(female)$ and hopefully $\pi(king) \approx \pi(queen)$ etc $\endgroup$ Nov 5 '17 at 20:55
  • $\begingroup$ Interesting! Another way saying saying how you're representing relations is something like pick a basis for $V$, compute the standard codimension-1 simplex, then $0$-cells represent words and $1$-cells represent relationships. $\endgroup$
    – zzz
    Nov 5 '17 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.