In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where $$J(\theta) = -\frac{1}{T}\sum_{t=1}^T\sum_{-m \leq j \leq m}\log p(w_{t+j}|w_t)$$ and $p(w_{t+j}|w_t) = \frac{exp(v_t\cdot u_{t+j}^T)}{\sum_{v \in V}exp(v_tu_v^T)}$ where $v_t$ represents the word embedding for middle word $w_t$ and $u_{t+j}$ represents word embedding for context word $w_{t+j}$.

Suppose we use stochastic gradient descent, then at every word $t$, we can find gradients with $\frac{\partial{J}}{\partial{v_t}}$, $\frac{\partial{J}}{\partial{u_1}}$, ... , $\frac{\partial{J}}{\partial{u_v}}$.

I see cross entropy being used as the objective function instead of the above $J(\theta)$. Why is it used and how does it come into play ? For example this stack exchange post talks about cross entropy loss for training word2vec.


1 Answer 1


In short, they're the same thing—up to a sign difference. Minimizing cross entropy is equivalent to minimizing the negative log-likelihood, which is equivalent to maximizing the likelihood.

I mean this in the most complete of senses: the formulas for cross entropy and average negative log-likelihood are identical. To avoid reinventing the wheel, I'll link to the Wikipedia article on cross-entropy. It outlines the equivalence formulaically.

The equation you've shown at the top, though, is slightly wrong—it's missing a $\log$. It should be a double summation of log-probabilities.

  • $\begingroup$ Hi yes you are right, I missed the log. Thanks for pointing ! $\endgroup$
    – calveeen
    Feb 24, 2021 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.