2
$\begingroup$

By default, word2vec uses 2 vectors for each word: one for the center word and one for the context word:

$\color{steelblue}{\large \text{Word2vec: objective function}}$

$\color{darkred}{\scriptstyle{\bullet}} \quad \text{We want to minimize the objective function:}$

$$J(\theta) = -\frac{1}{T} \sum_{t=1}^T \sum_\underset{{j \, \neq \, 0}}{\scriptsize{-m \leq j \leq m}} \log P(\mathbb{w}_{t+j} \,| \, \mathbb{w}_t; \theta)$$

$\color{darkred}{\scriptstyle{\bullet}} \quad \underline{\text{Question:}} \; \color{orchid}{\text{How to calculate } P(\mathbb{w}_{t+j} \,| \, \mathbb{w}_t; \theta) \text{?}}$

$\color{darkred}{\scriptstyle{\bullet}} \quad \underline{\text{Answer:}} \; \text{We will }\textit{use two } \text{vectors per } \mathbb{w}\text{:}$

$\quad \color{steelblue}{\scriptstyle{\bullet}} \quad v_\mathbb{w} \, \text{when }\mathbb{w} \,\text{is a center word}$

$ \quad \color{steelblue}{\scriptstyle{\bullet}} \quad u_\mathbb{w} \, \text{when }\mathbb{w} \,\text{is a context word}$

$\color{darkred}{\scriptstyle{\bullet}} \quad \text{Then for a center word } c \text{ and a context word }o\text{:}$

$$\color{Orchid}{P(o\,|\,c) = \frac{\exp u_o^T v_c}{\sum_{\mathbb{w} \in V} \exp u_\mathbb{w}^T v_c}}$$

(source, mirror)

How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?

$\endgroup$
3
  • 1
    $\begingroup$ I tried to convert your image to text as faithful as I could, I hope you don't mind. $\endgroup$ Dec 1, 2019 at 2:12
  • $\begingroup$ @FransRodenburg thanks very much! I'm curious, did you do it manually or use some program? $\endgroup$ Dec 1, 2019 at 2:58
  • 1
    $\begingroup$ You're welcome, I did it manually! $\endgroup$ Dec 1, 2019 at 11:29

1 Answer 1

0
$\begingroup$

See this post.

To summarize the post above, the goal of word2vec is to compute word embeddings such that semantically similar words are embedded closer to each other, and conversely, dissimilar words are embedded farther from each other.

In the neural network formulation, that means we minimize $$u_c^T v_c$$ because a word c is rarely in its own context. This minimization is not feasible if we constrain u to be same as v.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.