1
$\begingroup$

I am trying to understand how a Word2Vec is being trained. I understand that it can be trained using a CBOW and SkipGram. I am however lost as to what the probabilities are in the embedding layer.

enter image description here

Also what I do not understand from this image is, the input is "cat", let's say it is one-hot encoded by [1 0 0 0]. Where is the rest of this SkipGram part? Or is this what is in the embedding layer?

$\endgroup$

1 Answer 1

2
$\begingroup$

The values in the embedding layer are not probabilities. They are a high-dimensional representation of the word.

The way that Word2Vec is trained indeed focuses on having words that tend to co-occur be closer together in the embedding space, because the objective is equivalent, loosely, to minimizing the negative log-likelihood (NLL) of context words given a center word (skip-gram) or minimizing the NLL of a word given its context (CBOW). However, the probabilities in the objective and the numbers in the embedding are not the same. The values of the embedding vector are instead trained via some optimization algorithm (i.e. SGD) based on this probabilistic formulation.

I'm a bit confused by your second question about "the rest of the skip-gram part" -- without more context to the image I'm not sure I understand what you're asking.

$\endgroup$
3
  • $\begingroup$ What is meant by this high-dimensional representation of the word? And what I mean with the second part is, it uses SkipGram, but where is it used? Since the input is merely the word "cat"? Where does the training see which words are close to cat? $\endgroup$ Mar 29, 2020 at 11:22
  • $\begingroup$ This just means that each word is represented by a vector in high dimensional space. You can imagine in 3D, for example, a giant "word cloud" where related words are closer together. This is technically an oversimplification, but provides pretty good intuition. $\endgroup$ Mar 29, 2020 at 19:18
  • $\begingroup$ As to your second question -- in practice, for training Word2Vec we usually use negative sampling, so you don't explicitly see the SkipGram construction in the architecture of the embedding model. Generally, what happens is you end up training sort of a binary problem, where you create training examples by generating tuples of ((center_word, outside_word), 1) and ((center_word, random_word), 0). The original paper on this technique has much more details: papers.nips.cc/paper/… $\endgroup$ Mar 29, 2020 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.