I was reading the paper Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013 NIPS) and came across a part that I cannot quite understand.

Specifically, the part that I'm having trouble with is the beginning of page 3:

enter image description here

This is basically the softmax function that the basic Skip-gram model uses in order to compute the probabilities for each context word $w_{t + j}$ (where $-c\le j \le c$ with $c$ being the size of the context) given the target word $w_t$.

What I don't understand is the part in the screenshot where it uses the phrase "vector representations of $w$." Where do these vector representations come from? Are they a part of the Skip-gram model?

Perhaps my basic understanding of the Skip-gram model is incorrect, but I was under the impression that since Word2Vec may utilize the Skip-gram algorithm that it wouldn't contain vector representations.

In other words, I thought that the point of using the Skip-gram model was to receive a one-hot-encoded vector representing the target word and then output a vector containing the probabilities that words in the corpus are near this word, hence the vector representation. If the output of the Skip-gram model is such a vector representation, what are the vector representations being used within the Skip-gram model coming from?

Thank you.


The skip-gram model stores two vectors for each word $w$: $v_w$ and $v'_w$.

The result of word2vec training are the $v_w$ vectors. If you stack vectors $v'_w$, you get a projection matrix of a linear classifier.

Note that the $v_w$ vectors get updated whenever word $w$ occurs in the training data. On the other hand, $v'$ get updated when they are selected randomly with negative sampling which makes the output distribution estimation less reliable than standard softmax and it might be also the reason why vectors $v_w$ are used as the embeddings.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.