# Does the skipgram language model try to predict all context words at the same time?

In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation of a word is used as its vector representation.

I could now construct training examples by pairing current words with one of their surrounding words each. Each example would then be a pair of two one-hot encoded vectors, i.e. with all zeros except for one element. Here are some training examples:

$$[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0]\\ [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0]\\ [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]$$

Alternatively, I could sum up the surrounding words for each word. Each example would then consist of the one-hot encoded current word and a vector representing the surrounding words where some elements are one and most are zero. For the three examples above, there would be only one example in this case:

$$[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0]\\$$

Which representation makes more sense and why? I think the second one is more efficient and the only down-side I could see is that it cannot represent the same word occuring twice in the surrounding.

• Are you using one hot vectors as an example ? W2V uses dense vectors. For skip-gram, because of this dense representation, context words are predicted in a sequence and not at the same time. – Cedias May 21 '16 at 9:47
• @Cedias How to get the dense representations in the first place? I thought the idea was to convert one-hot vectors to dense vectors using this method. – danijar May 21 '16 at 9:49
• They are randomly initialized, then optimized with the skip-gram algorithm. – Cedias May 21 '16 at 9:55
• Thanks for the explanation, that makes sense. I'll have another look at the paper. Feel free to put this into a short answer. – danijar May 21 '16 at 10:08

## 1 Answer

It seems you have misunderstood how W2V algorithms work. Both W2V algorithms (Skip-Gram, Continuous BoW) use dense vectors initialized randomly which are optimized afterwards.

For skip-gram, because of this dense representation, context words are predicted in a sequence. (Christopher Moody does a great job explaining Skip-Gram here)

If you're familiar with C language you should read the original code here

Besides, Tensorflow documentation also provides a nice explanation of these models.