1
$\begingroup$

Two basic questions about word2vec.

  1. While training a skip-gram word2vec model, is the training data 1-to-1 or 1-to-many, i.e., say we have a sentence "the quick brown fox jumps over the lazy dog .." and we take a window size of 5, then would the training data using bag of words(BoW) model be of the form (as suggested by figure 1 in the paper https://arxiv.org/pdf/1301.3781.pdf),

(Note: for the purpose of understanding let the function BoW() denote the representation using a bag of words model.)

x = [

BoW(brown)

]

y = [

BoW(the, quick, fox, jumps)

]

or of the form (as suggested by the first image in http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/),

x = [

BoW(brown),

BoW(brown),

BoW(brown),

BoW(brown)

]

y = [

BoW(the),

BoW(quick),

BoW(fox),

BoW(jumps)

]

To further explain it clearly, should the output try to maximize the contexts as a set in former setup or maximize individual context word as in the later setup.

  1. Intuitively, it is easy to understand the skip-gram model, i.e. similar words would have similar contexts and hence the hidden layer should have similar weights to maximize the similar contexts. But it is not as intuitive to understand what a CBOW model is doing. How can I understand it in layman terms?
$\endgroup$
1
$\begingroup$

Skip Gram Model

While training a skip-gram word2vec model, is the training data 1-to-1 or 1-to-many, i.e., say we have a sentence "the quick brown fox jumps over the lazy dog .." and we take a window size of 5, then would the training data using bag of words(BoW) model be of the form (as suggested by figure 1 in the paper https://arxiv.org/pdf/1301.3781.pdf), or of the form (as suggested by the first image in http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/).

Before we talk about training I want to point out that once you have a trained skip-gram model the output of the model is a Vx1 vector containing probabilities, where V is the size of the vocabulary. Specifically, these are the probabilities of the context words, given the center word. During the training step, 1 output is produced (probabilities), but that single output is compared to multiple one hot encoded vectors (these are the context words) and the errors are summed and propagated back to update the model.

The training data shown in McCormick's tutorial is accurate with the caveat being that typically not all context words are included as training data. enter image description here

If you look at section 3.2 of the Google Paper you have referenced you'll noticed they write

Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples...Thus, if we choose C = 5, for each training word we will select randomly a number R in range < 1; C >, and then use R words from history and R words from the future of the current word as correct labels.

Meaning, in the McCormick example, rather than guaranteeing that we look at C words in the context, we will see up to C words.

To understand how the skip-gram model is updating its weights, it is helpful to go through the derivation of the update rules during back-prop. Here is a resource that goes through the back-prop math for both CBOW and Skip-Gram. You will see that that the update rule depends on comparing the one-hot encoded outputs of each context to the outputted probability distribution.

CBOW Model

In the continuous bag of words architecture, we have multiple inputs (i.e. the context words) and our single output is now the center word. During the forward pass of this model, each one-hot encoded context word is multiplied by the word embedding layer and all the context words are then averaged together. The intuition behind CBOW is similar to that of skip-gram in that similar words have similar context, except now rather than comparing one word to one output we are comparing the average of a group of words to one output.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.