How are word vector representations derived in Skip-gram?

Question

I was reading the paper Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al., 2013 NIPS) and came across a part that I cannot quite understand.

Specifically, the part that I'm having trouble with is the beginning of page 3:

This is basically the softmax function that the basic Skip-gram model uses in order to compute the probabilities for each context word $w_{t + j}$ (where $-c\le j \le c$ with $c$ being the size of the context) given the target word $w_t$.

What I don't understand is the part in the screenshot where it uses the phrase "vector representations of $w$." Where do these vector representations come from? Are they a part of the Skip-gram model?

Perhaps my basic understanding of the Skip-gram model is incorrect, but I was under the impression that since Word2Vec may utilize the Skip-gram algorithm that it wouldn't contain vector representations.

In other words, I thought that the point of using the Skip-gram model was to receive a one-hot-encoded vector representing the target word and then output a vector containing the probabilities that words in the corpus are near this word, hence the vector representation. If the output of the Skip-gram model is such a vector representation, what are the vector representations being used within the Skip-gram model coming from?

Thank you.

Answer 1

The skip-gram model stores two vectors for each word $w$: $v_w$ and $v'_w$.

The result of word2vec training are the $v_w$ vectors. If you stack vectors $v'_w$, you get a projection matrix of a linear classifier.

Note that the $v_w$ vectors get updated whenever word $w$ occurs in the training data. On the other hand, $v'$ get updated when they are selected randomly with negative sampling which makes the output distribution estimation less reliable than standard softmax and it might be also the reason why vectors $v_w$ are used as the embeddings.