2
$\begingroup$

We know that there are two different implementations of Word2Vec: CBOW and skip-gram. The following figure shows these implementations:

enter image description here

My question is about the skip-gram model. We expect that the training samples in the skip-gram model be like $input = w_i$ and $target = (w_{i-k},\ldots,w_{i-1},w_{i+1},\ldots,w_{i+k})$. But, in some implementations such as this one in tensorflow, the training samples are like $input=w_i$, $target=w_j$ for $w_j$ in the vicinity of $w_i$ (look at line 144, 145 of the code). This can be considered as the implementation of the following model, not the exact skip-gram model presented in the above picture.

enter image description here

Question 1: Why this is called as an implementation of skip-gram?

Question 2: With this implementation, why it is called that the skip-gram tries to predict the context from a particular target word, while we can simply say that it tries to predict a word from one of its neighbors?

$\endgroup$
2
$\begingroup$

Any code that iterates over 2*k target words, or 2*k context words, to create a total of 2*k (context-word)->(target-word) pairs for training, is "skip-gram". Some of the diagrams or notation in the original paper may give the impression skip-gram is using multiple context words at once, or predicting multiple target words at once, but in fact it's always just a 1-to-1 training pair, involving pairs-of-words in the same (window-sized) neighborhood.

(Only CBOW, which actually sums/averages multiple context words together, truly uses a combined range of ( w^(i-k)), ..., w^(i+k) ) words as a single NN-training example.)

If I recall correctly, the original word2vec paper described skip-gram in one way, but then at some point for CPU cache efficiency the Google-released word2vec.c code looped over the text in the opposite way – which has sometimes caused confusion for people reading that code, or other code modeled on it.

But whether you view skip-gram as predicting a central target word from individual nearby context words, or as predicting surrounding individual target words from a central context word, in the end each original text sample results in the exact same set of desired (context-word)->(target-word) predictions – just in a slightly different training order. Each ordering is reasonably called 'skip-gram' and winds up with similar results, at the end of bulk training.

$\endgroup$
  • $\begingroup$ Thanks for your comment. But I think my questions are misunderstood. In addition, "predicting a central target word from nearby context words" presents CBOW model not skip-gram. $\endgroup$ – Hossein Jun 12 '17 at 10:37
  • $\begingroup$ I mean in the skip-gram model, the target should be (wi−k,…,wi−1,wi+1,…,wi+k) not just a single word. $\endgroup$ – Hossein Jun 12 '17 at 10:59
  • $\begingroup$ Yes, and I'm saying it doesn't matter if you loop through a text setting the (nn input) context to be w^i, then the (nn output) targets to be ( w^(i-k)), ..., w^(i+k) ), or if you loop through the text setting the (nn output) target to be w^i, and the (nn input) contexts to be ( w^(i-k)), ..., w^(i+k) ). In both looping strategies, every word is used to predict every within-window neighbor – the exact same (context-word)->(target-word) pairs are neural-network trained – just in a slightly different order. Hence both ways of enumerating the training-pairs are "skip-gram". $\endgroup$ – gojomo Jun 12 '17 at 18:04
  • 2
    $\begingroup$ You're misreading the paper's notation & my answers. Any code that iterates over 2*k target words, or 2*k context words, to create 2*k context->target pairs for training, is "skip-gram". (Only if the 2*k context words are summed/averaged is it then CBOW.) So when you see Google's word2vec.c, or gensim's word2vec.py, or other implementations, & their 'skip-gram' seems a little different than what's described in the paper – the code is correct, the paper means 'each word in turn', & different ways of looping are superficial changes that keep the essence of 'skip-gram' in the end-result. $\endgroup$ – gojomo Jun 16 '17 at 16:47
  • 1
    $\begingroup$ Thanks. Your last comment is the answer to my question. I would like to accept the answer if you modify it to this comment. $\endgroup$ – Hossein Jun 16 '17 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.