1
$\begingroup$

I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4):

$$ p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})}{\sum_{j'=1}^{V}\exp(\mathbf{ v’}^{T}_{w_{j'}} \mathbf{ v}_{w_{I}})} $$ Simply said, do I understand correctly that wj describes my target word and wi is the single context word?

Would the equation look different with multiple context words or does it remain the same because wi describes the average of all context words?

Edit: Added the equation in TeX

$\endgroup$
0

1 Answer 1

2
$\begingroup$

You’ve correctly understood $w_j$ and $w_I$: they’re the target word and input word.

Section 1.2 goes into more detail about the CBOW case; I’m filling in the pieces that weren’t explicit.

The LHS of the equation would be different: $$p(w_j \mid w_{I1}, \ldots, w_{IC})$$ to show that you’re conditioning on several input words.

The RHS is also different, because we no longer use $\mathbf{v}_{w_I}$ to represent the input. Instead, the numerator and denominator should both replace that term with $\bf h$ from equation (18):

$$\mathbf{h} = \frac{1}{C} \left(\mathbf{v}_{w_{I1}} + \mathbf{v}_{w_{I2}} + \ldots + \mathbf{v}_{w_{IC}} \right)^T$$

This term averages the representations of the $C$ input words.

$\endgroup$
4
  • $\begingroup$ Thank you for your quick response. I'm at a loss here. How would the final equation look like? Do I insert $h$ into $v_{w_{I}}$ of (4)? $\endgroup$
    – jonas
    Apr 6, 2021 at 23:33
  • $\begingroup$ Yep! The results of that derivation are equations 19–21 in the paper. Another way to get the expression you’re looking for is to negate, then exponentiate, equation 19. $\endgroup$ Apr 6, 2021 at 23:39
  • $\begingroup$ I know that I've already accepted your answer but I have one follow up question regarding equation 19-21. Am I understanding it correctly that $w_{o}$ is my actual target word from the vocabulary $V$ and $j*$ is the corresponding index in the output layer. Thanks again for helping me to understand it in depth, you can't imagine how grateful I am. $\endgroup$
    – jonas
    Apr 7, 2021 at 14:18
  • $\begingroup$ Yup, that’s what both are! They’re defined on page 3. $\endgroup$ Apr 7, 2021 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.