I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice.

The gradient in question is here. The video should already be at the right timestamp. Apparently this is a common format usually comes from a softmax-style problem.

I more or less see how this was derived, but for some reason am failing to see how this would succeed in updating embeddings with $d$ dimensions.

My understanding

My understanding is that both $U$ and $V$ are matrices of size $d$ x $|V|$. Both contain a representation of all vocabulary words, but one is a representation for the word as part of the context, and the other is a representation for the word when it is the center word.

We really need to take two derivatives here, then:

  1. $\frac{d}{dv_c}$
  2. $\frac{d}{du_c}$

And during each update step, we'll want to update each column of V using 1., and each column of U using 2. Let's take a look at the first of the two gradient update functions, as derived in class:

$\frac{d}{dv_c} = u_0 - \sum_{x=1}^{v}p(x|c)u_x$.

My general confusion

As an exercise for myself, I wanted to work out an example "on paper" that applies this update using gradient descent.

But I'm failing to see how this would actually work. $u_0$ is apparently the "what we observed: the output context word appeared" to quote the video exactly. I don't follow that. Aren't all indices of $U$ supposed to be the $d$ dimensional word embeddings? I thought those were inputs.

The summation I imagine would involve looking through all of the training data to count the actual probabilities. But then we're multiplying by $u_x$ for each word in the vocabulary.

Finally, suppose all word embeddings are initialized to zeros. Then, clearly my understanding is wrong, because the embeddings would never leave zero using this logic.

I guess what I'm trying to ask

What exactly is $u_0$ and $u_x$, then? How can I use them during the update step to change my word embeddings, which could be initialized to whatever (say all zeros), to the appropriate word vectors?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.