I am trying to work my way through the first problem set of the cs224d online stanford class course material and I am having some issues with problem 3A: When using the skip gram word2vec model with the softmax prediction function and the cross entropy loss function, we want to calculate the gradients with respect to the predicted word vectors. So given the softmax function:

$ \hat{w_i} = \Pr(word_i\mid\hat{r}, w) = \frac{\exp(w_i^T \hat{r})}{\sum_{j}^{|V|}exp(w_j^T\hat{r})}$

and cross entropy function:

$CE(w, \hat{w}) = -\sum\nolimits_{k} w_klog(\hat{w_k})$

we need to calculate $\frac{\partial{CE}}{\partial{\hat{r}}}$

My steps are as follows:

$CE(w, \hat{w}) = -\sum_{k}^{|V|} w_klog(\frac{\exp(w_k^T \hat{r})}{\sum_{j}^{|V|}exp(w_j^T\hat{r})})$

$= -\sum_{k}^{|V|} w_klog(\exp(w_k^T \hat{r}) - w_klog(\sum_{j}^{|V|}exp(w_j^T\hat{r}))$

now given $w_k$ is a one hot vector and i is the correct class:

$CE(w, \hat{w}) = - w_i^T\hat{r} + log(\sum_{j}^{|V|}exp(w_j^T\hat{r}))$

$\frac{\partial{CE}}{\partial{\hat{r}}} = -w_i + \frac{1}{\sum_{j}^{|V|}exp(w_j^T\hat{r})}\sum_{j}^{|V|}exp(w_j^T\hat{r})w_j$

Is this correct or could it be simplified further? I want to try to make sure I am on the right track as the problem set solutions aren't posted online. Plus getting the written assignments correct are important to being able to properly do the programming assignments.

  • $\begingroup$ Please add the self-study tag to the question $\endgroup$
    – Dawny33
    Commented Sep 22, 2015 at 7:21
  • $\begingroup$ 2nd minus sign in first log identity should be a plus. Tried to fix it for you but edits need to be at least 6 chars :\ $\endgroup$
    – FatalMojo
    Commented Sep 22, 2015 at 8:41

1 Answer 1


$$\frac{\partial{CE}}{\partial{\hat{r}}} = -w_i + \frac{1}{\sum_{j}^{|V|}exp(w_j^T\hat{r})}\sum_{j}^{|V|}exp(w_j^T\hat{r})w_j$$ can be rewritten as $$\frac{\partial{CE}}{\partial{\hat{r}}} = -w_i + \sum_{j}^{|V|} \left( \frac{ \exp(w_j^\top\hat{r}) }{\sum_{j}^{|V|}exp(w_j^T\hat{r})} \cdot w_j \right)$$ note, the sums are both indexed by j but it really should be 2 different variables. This would be more appropriate $$\frac{\partial{CE}}{\partial{\hat{r}}} = -w_i + \sum_{x}^{|V|} \left( \frac{ \exp(w_x^\top\hat{r}) }{\sum_{j}^{|V|}exp(w_j^T\hat{r})} \cdot w_x \right)$$ which translates to $$\frac{\partial{CE}}{\partial{\hat{r}}} = -w_i + \sum_{x}^{|V|} \Pr(word_x\mid\hat{r}, w) \cdot w_x$$

  • 1
    $\begingroup$ Relevant, he goes over that derivation in details in lecture 2 @ 38:00 $\endgroup$
    – FatalMojo
    Commented Sep 22, 2015 at 8:24
  • $\begingroup$ Why should the sums be indexed by different variables? $\endgroup$
    – Yamaneko
    Commented Sep 29, 2015 at 16:19
  • 1
    $\begingroup$ Just to avoid confusion. Mathematically it means the same thing, but it's good practice to change the index label when adding a new sum. $\endgroup$
    – FatalMojo
    Commented Sep 30, 2015 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.