Skip Gram Model
While training a skip-gram word2vec model, is the training data 1-to-1 or 1-to-many, i.e., say we have a sentence "the quick brown fox jumps over the lazy dog .." and we take a window size of 5, then would the training data using bag of words(BoW) model be of the form (as suggested by figure 1 in the paper https://arxiv.org/pdf/1301.3781.pdf), or of the form (as suggested by the first image in http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/).
Before we talk about training I want to point out that once you have a trained skip-gram model the output of the model is a Vx1 vector containing probabilities, where V is the size of the vocabulary. Specifically, these are the probabilities of the context words, given the center word. During the training step, 1 output is produced (probabilities), but that single output is compared to multiple one hot encoded vectors (these are the context words) and the errors are summed and propagated back to update the model.
The training data shown in McCormick's tutorial is accurate with the caveat being that typically not all context words are included as training data.
If you look at section 3.2 of the Google Paper you have referenced you'll noticed they write
Since the more distant words are usually less related to the current
word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples...Thus, if we choose C = 5, for each training word we will select randomly a number R in range < 1; C >, and then use R words from history and R words from the future of the current word as correct labels.
Meaning, in the McCormick example, rather than guaranteeing that we look at C words in the context, we will see up to C words.
To understand how the skip-gram model is updating its weights, it is helpful to go through the derivation of the update rules during back-prop. Here is a resource that goes through the back-prop math for both CBOW and Skip-Gram. You will see that that the update rule depends on comparing the one-hot encoded outputs of each context to the outputted probability distribution.
In the continuous bag of words architecture, we have multiple inputs (i.e. the context words) and our single output is now the center word. During the forward pass of this model, each one-hot encoded context word is multiplied by the word embedding layer and all the context words are then averaged together. The intuition behind CBOW is similar to that of skip-gram in that similar words have similar context, except now rather than comparing one word to one output we are comparing the average of a group of words to one output.