Do we use maximum likelihood or cross entropy Loss for training skip-gram model?

Question

In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where $$J(\theta) = -\frac{1}{T}\sum_{t=1}^T\sum_{-m \leq j \leq m}\log p(w_{t+j}|w_t)$$ and $p(w_{t+j}|w_t) = \frac{exp(v_t\cdot u_{t+j}^T)}{\sum_{v \in V}exp(v_tu_v^T)}$ where $v_t$ represents the word embedding for middle word $w_t$ and $u_{t+j}$ represents word embedding for context word $w_{t+j}$.

Suppose we use stochastic gradient descent, then at every word $t$, we can find gradients with $\frac{\partial{J}}{\partial{v_t}}$, $\frac{\partial{J}}{\partial{u_1}}$, ... , $\frac{\partial{J}}{\partial{u_v}}$.

I see cross entropy being used as the objective function instead of the above $J(\theta)$. Why is it used and how does it come into play ? For example this stack exchange post talks about cross entropy loss for training word2vec.

Answer 1

In short, they're the same thing—up to a sign difference. Minimizing cross entropy is equivalent to minimizing the negative log-likelihood, which is equivalent to maximizing the likelihood.

I mean this in the most complete of senses: the formulas for cross entropy and average negative log-likelihood are identical, except that one is divided by the number of examples. To avoid reinventing the wheel, I'll link to the Wikipedia article on cross-entropy. It outlines the equivalence formulaically.

~~The equation you've shown at the top, though, is slightly wrong—it's missing a $\log$. It should be a double summation of log-probabilities.~~ As of Feb 24, 2021, the OP has fixed the formula.