3
$\begingroup$

We know that there are two different implementations of Word2Vec: CBOW and skip-gram. The following figure shows these implementations:

enter image description here

My question is about the skip-gram model. We expect that the training samples in the skip-gram model be like $input = w_i$ and $target = (w_{i-k},\ldots,w_{i-1},w_{i+1},\ldots,w_{i+k})$. But, in some implementations such as this one in tensorflow, the training samples are like $input=w_i$, $target=w_j$ for $w_j$ in the vicinity of $w_i$ (look at line 144, 145 of the code). This can be considered as the implementation of the following model, not the exact skip-gram model presented in the above picture.

enter image description here

Question 1: Why this is called as an implementation of skip-gram?

Question 2: With this implementation, why it is called that the skip-gram tries to predict the context from a particular target word, while we can simply say that it tries to predict a word from one of its neighbors?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Any code that iterates over 2*k target words, or 2*k context words, to create a total of 2*k (context-word)->(target-word) pairs for training, is "skip-gram". Some of the diagrams or notation in the original paper may give the impression skip-gram is using multiple context words at once, or predicting multiple target words at once, but in fact it's always just a 1-to-1 training pair, involving pairs-of-words in the same (window-sized) neighborhood.

(Only CBOW, which actually sums/averages multiple context words together, truly uses a combined range of ( w^(i-k)), ..., w^(i+k) ) words as a single NN-training example.)

If I recall correctly, the original word2vec paper described skip-gram in one way, but then at some point for CPU cache efficiency the Google-released word2vec.c code looped over the text in the opposite way – which has sometimes caused confusion for people reading that code, or other code modeled on it.

But whether you view skip-gram as predicting a central target word from individual nearby context words, or as predicting surrounding individual target words from a central context word, in the end each original text sample results in the exact same set of desired (context-word)->(target-word) predictions – just in a slightly different training order. Each ordering is reasonably called 'skip-gram' and winds up with similar results, at the end of bulk training.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for your comment. But I think my questions are misunderstood. In addition, "predicting a central target word from nearby context words" presents CBOW model not skip-gram. $\endgroup$
    – Hossein
    Jun 12, 2017 at 10:37
  • 1
    $\begingroup$ Yes, and I'm saying it doesn't matter if you loop through a text setting the (nn input) context to be w^i, then the (nn output) targets to be ( w^(i-k)), ..., w^(i+k) ), or if you loop through the text setting the (nn output) target to be w^i, and the (nn input) contexts to be ( w^(i-k)), ..., w^(i+k) ). In both looping strategies, every word is used to predict every within-window neighbor – the exact same (context-word)->(target-word) pairs are neural-network trained – just in a slightly different order. Hence both ways of enumerating the training-pairs are "skip-gram". $\endgroup$
    – gojomo
    Jun 12, 2017 at 18:04
  • 1
    $\begingroup$ The first one, where w^i is input and ( w^(i-k)), ..., w^(i+k) ) is output is the skip-gram model. The second one, where ( w^(i-k)), ..., w^(i+k) ) is input and w^i is output, is the CBOW model. These two are different. In the CBOW, all context words are used together to predict the center word, but in the skip-gram, the center word is used to predict the context words. My question is about skip-gram and why instead of all context words, just one context word is used as the target. $\endgroup$
    – Hossein
    Jun 16, 2017 at 9:41
  • 4
    $\begingroup$ You're misreading the paper's notation & my answers. Any code that iterates over 2*k target words, or 2*k context words, to create 2*k context->target pairs for training, is "skip-gram". (Only if the 2*k context words are summed/averaged is it then CBOW.) So when you see Google's word2vec.c, or gensim's word2vec.py, or other implementations, & their 'skip-gram' seems a little different than what's described in the paper – the code is correct, the paper means 'each word in turn', & different ways of looping are superficial changes that keep the essence of 'skip-gram' in the end-result. $\endgroup$
    – gojomo
    Jun 16, 2017 at 16:47
  • 2
    $\begingroup$ Thanks. Your last comment is the answer to my question. I would like to accept the answer if you modify it to this comment. $\endgroup$
    – Hossein
    Jun 16, 2017 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.