1
$\begingroup$

I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.

1 - Taking a look at this blog post and Andrew Ng's Deep Learning Specialization, I understood that, for each word, we generate one training sample for each context word.

So, if we have the sentence "cat sat on the mat" we will have samples:

(cat, sat) (sat, cat) (sat, on)

And so on.

Then you train your network which will have the dimensions.

  • Input: $(V,1)$
  • Weights 1: $(d,V)$
  • Hidden layer: $(d,1)$
  • Weights 2: $(V,d)$
  • Output layer: $(V,1)$

Ok, so these dimensions match and we are good. For one given input, given that the weights are the same, we always have the same estimation in the output layer.

In this definition we have symmetric samples (for example, (cat, sat) and (sat, cat)) and saying that we use center words as inputs and context words as outputs is meaningless since they are interchangeable?

2 - Watching Stanford's NLP with Deep Learning class (38:54), it seems like for the same center word we can get different outputs:

enter image description here

The numbers in the red circles that I drew should be the same from what I understand. I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer.

3- I saw in other places (couldn't find reference now) another formulation: for each input word, we represent the context as a vector with ones for the context words and zeros elsewhere. So, in this case, instead of one training example for each context word, we would have one training example for each center word.

$\endgroup$
0
$\begingroup$

I think that this graph is incorrect, output of hidden layer is the same for each panel. we can finally compute the output of each node via the softmax function. Thus, we will generate C identical vectors V-dims. Finally, use the actual output to compute the loss function to optimize the word vector matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.