I'm using this source to understand the skip gram model. Let's say the context size is $4$ ($2$ context words on each side of the target word). This image illustrates how training examples are generated:

enter image description here

and this is in agreement with a few other sources and SE answers I've looked up. I'm also reading these notes which suggest the following architecture for skip gram:

enter image description here

This architecture suggests that a single training example must have the target word and all the context words in the window (e.g. (quick, brown, jumps, over, fox). The loss function as described in the lecture is (here the target word is indexed as $0$):

$$J=-\log P(w_{-2}, w_{-1}, w_1, w_2 | w_0) \\=-\log \prod_{i=-2,i\neq 0}^2 P(u_i | v_0) \\=-\log \prod_{i=-2,i\neq 0}^2 \frac{\exp(u_i^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$

Assuming the window here is $w_{-2}$: "quick", $w_{-1}$: "brown", $w_1$: "jumps", $w_2$: "over", with target word $w_0$: "fox",

  1. Is the neural network architecture correct? Based on the first link I posted, each training example has only one context word and so the last layer should only have $V$ nodes instead of $C\times V$.

  2. Accordingly, the loss function for each training example (say $(w_1, w_0)$) should be:

$$L(w_1, w_0) = -\log \frac{\exp(u_1^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$

Unlike a standard neural network, in which we form the cost function by taking the average of loss function over all training examples and then backpropagate on the basis of that, in this case we backpropagate with respect to the above loss function alone. We then move on to the next training example $w_2$, form a new loss function $L(w_2, w_0)$, backpropagate w.r.t. that, and so on. Is my understanding correct? Is the loss function as I described above correct? The other alternative I can think of is to form the cost function for that particular window by summing over loss functions for that window ($J=L(w_{-2},w_0)+L(w_{-1},w_0)+L(w_1,w_0)+L(w_2,w_0)$) and then doing backpropagation w.r.t. $J$.

  1. I've read that skip gram gives more weightage to nearby words rather than distant ones. I'm not sure why that's true. Given the window size of $2$ (say "quick brown fox jumps over"), it seems "quick" would be treated equivalently to "brown".

1 - The architecture in the CS224n course lecture notes is correct. The likelihood is given by the product of the probabilities $ \Pi_{w\in \rm{Text}} \Pi_{c \in C(w)} P(c | w)$ (where $C(w)$ is the context of the target word). Note that I have added the product over all the words in the corpus (see details https://arxiv.org/abs/1402.3722). Taking the log of the likelihood to define the loss function, you end up with a sum over all the target words.

Your confusion with the first source is related to your comment "Based on the first link I posted, each training example has only one context word". What they mean by having "training examples" of the form (target, context) such as (The, quick)... is that the likelihood decomposes into these $P(c | w)$ terms.

2- As commented previously, the loss function is a sum over all the target words (and for each target word a sum over its context) so this should solve your confusion regarding "Unlike a standard neural network, in which we form the cost function by taking the average of loss function over all training examples...".

3- agree with you, can you give more details?

  • $\begingroup$ Can you clarify one more thing - so I have a sliding window of $5$ words. For each such window, we formulate a cost function $J$ only for that window, do a backprop step with respect to that $J$ and then slide the window forward in the text. Is that correct? I'm asking this because in a standard neural network, the cost function is formulated by summing over loss functions of all training examples and then backprop is done. But in skip gram, it seems instead of the entire set of words, the cost function is defined only w.r.t the words in the window and the rest are ignored. $\endgroup$ – Shirish Kulhari Sep 2 at 7:47
  • $\begingroup$ - " Is that correct?" yes, thats how is stated in the CS224n notes, and at the end you sum all the contributions. - "... and then backprop is done." Given that the cost function is a sum over training examples, it is equivalent to calculating backprop for each training example and then adding the contributions no? (example brilliant.org/wiki/backpropagation). - "...the rest are ignored." in skip-gram there is also a global cost function, which decomposes as a sums over each target word in the corpus (with an inner sum over the corresponding context for each target) $\endgroup$ – Alejandro Celis Sep 3 at 5:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.