Confusion on how skip gram implementation is formulated I'm using this source to understand the skip gram model. Let's say the context size is $4$ ($2$ context words on each side of the target word). This image illustrates how training examples are generated:

and this is in agreement with a few other sources and SE answers I've looked up. I'm also reading these notes which suggest the following architecture for skip gram:

This architecture suggests that a single training example must have the target word and all the context words in the window (e.g. (quick, brown, jumps, over, fox). The loss function as described in the lecture is (here the target word is indexed as $0$):
$$J=-\log P(w_{-2}, w_{-1}, w_1, w_2 | w_0)
\\=-\log \prod_{i=-2,i\neq 0}^2 P(u_i | v_0)
\\=-\log \prod_{i=-2,i\neq 0}^2 \frac{\exp(u_i^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$
Assuming the window here is $w_{-2}$: "quick", $w_{-1}$: "brown", $w_1$: "jumps", $w_2$: "over", with target word $w_0$: "fox",


*

*Is the neural network architecture correct? Based on the first link I posted, each training example has only one context word and so the last layer should only have $V$ nodes instead of $C\times V$.

*Accordingly, the loss function for each training example (say $(w_1, w_0)$) should be:
$$L(w_1, w_0) = -\log \frac{\exp(u_1^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$
Unlike a standard neural network, in which we form the cost function by taking the average of loss function over all training examples and then backpropagate on the basis of that, in this case we backpropagate with respect to the above loss function alone. We then move on to the next training example $w_2$, form a new loss function $L(w_2, w_0)$, backpropagate w.r.t. that, and so on. Is my understanding correct? Is the loss function as I described above correct? The other alternative I can think of is to form the cost function for that particular window by summing over loss functions for that window ($J=L(w_{-2},w_0)+L(w_{-1},w_0)+L(w_1,w_0)+L(w_2,w_0)$) and then doing backpropagation w.r.t. $J$.


*I've read that skip gram gives more weightage to nearby words rather than distant ones. I'm not sure why that's true. Given the window size of $2$ (say "quick brown fox jumps over"), it seems "quick" would be treated equivalently to "brown".

 A: 1 - The architecture in the CS224n course lecture notes is correct.  The likelihood is given by the product of the probabilities $ \Pi_{w\in \rm{Text}} \Pi_{c \in C(w)} P(c | w)$ (where $C(w)$ is the context of the target word).  Note that I have added the product over all the words in the corpus (see details https://arxiv.org/abs/1402.3722).   Taking the log of the likelihood to define the loss function, you end up with a sum over all the target words.  
Your confusion with the first source is related to your comment "Based on the first link I posted, each training example has only one context word".   What they mean by having "training examples" of the form (target, context) such as (The, quick)... is that the likelihood decomposes into these $P(c | w)$ terms.   
2- As commented previously, the loss function is a sum over all the target words (and for each target word a sum over its context) so this should solve your confusion regarding "Unlike a standard neural network, in which we form the cost function by taking the average of loss function over all training examples...". 
3-  agree with you, can you give more details?  
A: I also had similar confusion in the past so I made a skip-gram model demo in this GitHub repo in javascript. Hopefully, it can help people to understand how the skip-gram model works by visualizing it.

