# How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?

By default, word2vec uses 2 vectors for each word: one for the center word and one for the context word:

$$\color{steelblue}{\large \text{Word2vec: objective function}}$$

$$\color{darkred}{\scriptstyle{\bullet}} \quad \text{We want to minimize the objective function:}$$

$$J(\theta) = -\frac{1}{T} \sum_{t=1}^T \sum_\underset{{j \, \neq \, 0}}{\scriptsize{-m \leq j \leq m}} \log P(\mathbb{w}_{t+j} \,| \, \mathbb{w}_t; \theta)$$

$$\color{darkred}{\scriptstyle{\bullet}} \quad \underline{\text{Question:}} \; \color{orchid}{\text{How to calculate } P(\mathbb{w}_{t+j} \,| \, \mathbb{w}_t; \theta) \text{?}}$$

$$\color{darkred}{\scriptstyle{\bullet}} \quad \underline{\text{Answer:}} \; \text{We will }\textit{use two } \text{vectors per } \mathbb{w}\text{:}$$

$$\quad \color{steelblue}{\scriptstyle{\bullet}} \quad v_\mathbb{w} \, \text{when }\mathbb{w} \,\text{is a center word}$$

$$\quad \color{steelblue}{\scriptstyle{\bullet}} \quad u_\mathbb{w} \, \text{when }\mathbb{w} \,\text{is a context word}$$

$$\color{darkred}{\scriptstyle{\bullet}} \quad \text{Then for a center word } c \text{ and a context word }o\text{:}$$

$$\color{Orchid}{P(o\,|\,c) = \frac{\exp u_o^T v_c}{\sum_{\mathbb{w} \in V} \exp u_\mathbb{w}^T v_c}}$$

(source, mirror)

How does using the same vector for the center word and for the context word impact the performance of word vectors in word2vec?

• I tried to convert your image to text as faithful as I could, I hope you don't mind. Dec 1, 2019 at 2:12
• @FransRodenburg thanks very much! I'm curious, did you do it manually or use some program? Dec 1, 2019 at 2:58
• You're welcome, I did it manually! Dec 1, 2019 at 11:29

In the neural network formulation, that means we minimize $$u_c^T v_c$$ because a word c is rarely in its own context. This minimization is not feasible if we constrain u to be same as v.