# Confusion on how skip gram implementation is formulated

I'm using this source to understand the skip gram model. Let's say the context size is $$4$$ ($$2$$ context words on each side of the target word). This image illustrates how training examples are generated: and this is in agreement with a few other sources and SE answers I've looked up. I'm also reading these notes which suggest the following architecture for skip gram: This architecture suggests that a single training example must have the target word and all the context words in the window (e.g. (quick, brown, jumps, over, fox). The loss function as described in the lecture is (here the target word is indexed as $$0$$):

$$J=-\log P(w_{-2}, w_{-1}, w_1, w_2 | w_0) \\=-\log \prod_{i=-2,i\neq 0}^2 P(u_i | v_0) \\=-\log \prod_{i=-2,i\neq 0}^2 \frac{\exp(u_i^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$

Assuming the window here is $$w_{-2}$$: "quick", $$w_{-1}$$: "brown", $$w_1$$: "jumps", $$w_2$$: "over", with target word $$w_0$$: "fox",

1. Is the neural network architecture correct? Based on the first link I posted, each training example has only one context word and so the last layer should only have $$V$$ nodes instead of $$C\times V$$.

2. Accordingly, the loss function for each training example (say $$(w_1, w_0)$$) should be:

$$L(w_1, w_0) = -\log \frac{\exp(u_1^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$

Unlike a standard neural network, in which we form the cost function by taking the average of loss function over all training examples and then backpropagate on the basis of that, in this case we backpropagate with respect to the above loss function alone. We then move on to the next training example $$w_2$$, form a new loss function $$L(w_2, w_0)$$, backpropagate w.r.t. that, and so on. Is my understanding correct? Is the loss function as I described above correct? The other alternative I can think of is to form the cost function for that particular window by summing over loss functions for that window ($$J=L(w_{-2},w_0)+L(w_{-1},w_0)+L(w_1,w_0)+L(w_2,w_0)$$) and then doing backpropagation w.r.t. $$J$$.

1. I've read that skip gram gives more weightage to nearby words rather than distant ones. I'm not sure why that's true. Given the window size of $$2$$ (say "quick brown fox jumps over"), it seems "quick" would be treated equivalently to "brown".

1 - The architecture in the CS224n course lecture notes is correct. The likelihood is given by the product of the probabilities $$\Pi_{w\in \rm{Text}} \Pi_{c \in C(w)} P(c | w)$$ (where $$C(w)$$ is the context of the target word). Note that I have added the product over all the words in the corpus (see details https://arxiv.org/abs/1402.3722). Taking the log of the likelihood to define the loss function, you end up with a sum over all the target words.
Your confusion with the first source is related to your comment "Based on the first link I posted, each training example has only one context word". What they mean by having "training examples" of the form (target, context) such as (The, quick)... is that the likelihood decomposes into these $$P(c | w)$$ terms.
• Can you clarify one more thing - so I have a sliding window of $5$ words. For each such window, we formulate a cost function $J$ only for that window, do a backprop step with respect to that $J$ and then slide the window forward in the text. Is that correct? I'm asking this because in a standard neural network, the cost function is formulated by summing over loss functions of all training examples and then backprop is done. But in skip gram, it seems instead of the entire set of words, the cost function is defined only w.r.t the words in the window and the rest are ignored. – Shirish Kulhari Sep 2 at 7:47