# Why the skip-gram model is called as predicting source context words from the target word?

We know that there are two different implementations of Word2Vec: CBOW and skip-gram. The following figure shows these implementations:

My question is about the skip-gram model. We expect that the training samples in the skip-gram model be like $$input = w_i$$ and $$target = (w_{i-k},\ldots,w_{i-1},w_{i+1},\ldots,w_{i+k})$$. But, in some implementations such as this one in tensorflow, the training samples are like $$input=w_i$$, $$target=w_j$$ for $$w_j$$ in the vicinity of $$w_i$$ (look at line 144, 145 of the code). This can be considered as the implementation of the following model, not the exact skip-gram model presented in the above picture.

Question 1: Why this is called as an implementation of skip-gram?

Question 2: With this implementation, why it is called that the skip-gram tries to predict the context from a particular target word, while we can simply say that it tries to predict a word from one of its neighbors?

Any code that iterates over 2*k target words, or 2*k context words, to create a total of 2*k (context-word)->(target-word) pairs for training, is "skip-gram". Some of the diagrams or notation in the original paper may give the impression skip-gram is using multiple context words at once, or predicting multiple target words at once, but in fact it's always just a 1-to-1 training pair, involving pairs-of-words in the same (window-sized) neighborhood.

(Only CBOW, which actually sums/averages multiple context words together, truly uses a combined range of ( w^(i-k)), ..., w^(i+k) ) words as a single NN-training example.)

If I recall correctly, the original word2vec paper described skip-gram in one way, but then at some point for CPU cache efficiency the Google-released word2vec.c code looped over the text in the opposite way – which has sometimes caused confusion for people reading that code, or other code modeled on it.

But whether you view skip-gram as predicting a central target word from individual nearby context words, or as predicting surrounding individual target words from a central context word, in the end each original text sample results in the exact same set of desired (context-word)->(target-word) predictions – just in a slightly different training order. Each ordering is reasonably called 'skip-gram' and winds up with similar results, at the end of bulk training.

• Thanks for your comment. But I think my questions are misunderstood. In addition, "predicting a central target word from nearby context words" presents CBOW model not skip-gram. Jun 12, 2017 at 10:37
• Yes, and I'm saying it doesn't matter if you loop through a text setting the (nn input) context to be w^i, then the (nn output) targets to be ( w^(i-k)), ..., w^(i+k) ), or if you loop through the text setting the (nn output) target to be w^i, and the (nn input) contexts to be ( w^(i-k)), ..., w^(i+k) ). In both looping strategies, every word is used to predict every within-window neighbor – the exact same (context-word)->(target-word) pairs are neural-network trained – just in a slightly different order. Hence both ways of enumerating the training-pairs are "skip-gram". Jun 12, 2017 at 18:04
• The first one, where w^i is input and ( w^(i-k)), ..., w^(i+k) ) is output is the skip-gram model. The second one, where ( w^(i-k)), ..., w^(i+k) ) is input and w^i is output, is the CBOW model. These two are different. In the CBOW, all context words are used together to predict the center word, but in the skip-gram, the center word is used to predict the context words. My question is about skip-gram and why instead of all context words, just one context word is used as the target. Jun 16, 2017 at 9:41
• You're misreading the paper's notation & my answers. Any code that iterates over 2*k target words, or 2*k context words, to create 2*k context->target pairs for training, is "skip-gram". (Only if the 2*k context words are summed/averaged is it then CBOW.) So when you see Google's word2vec.c, or gensim's word2vec.py, or other implementations, & their 'skip-gram' seems a little different than what's described in the paper – the code is correct, the paper means 'each word in turn', & different ways of looping are superficial changes that keep the essence of 'skip-gram' in the end-result. Jun 16, 2017 at 16:47
• Thanks. Your last comment is the answer to my question. I would like to accept the answer if you modify it to this comment. Jun 16, 2017 at 18:53