1
$\begingroup$

In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation of a word is used as its vector representation.

I could now construct training examples by pairing current words with one of their surrounding words each. Each example would then be a pair of two one-hot encoded vectors, i.e. with all zeros except for one element. Here are some training examples:

$$ [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0]\\ [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0]\\ [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0] $$

Alternatively, I could sum up the surrounding words for each word. Each example would then consist of the one-hot encoded current word and a vector representing the surrounding words where some elements are one and most are zero. For the three examples above, there would be only one example in this case:

$$ [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0] \rightarrow [0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0]\\ $$

Which representation makes more sense and why? I think the second one is more efficient and the only down-side I could see is that it cannot represent the same word occuring twice in the surrounding.

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you using one hot vectors as an example ? W2V uses dense vectors. For skip-gram, because of this dense representation, context words are predicted in a sequence and not at the same time. $\endgroup$
    – Cedias
    Commented May 21, 2016 at 9:47
  • $\begingroup$ @Cedias How to get the dense representations in the first place? I thought the idea was to convert one-hot vectors to dense vectors using this method. $\endgroup$
    – danijar
    Commented May 21, 2016 at 9:49
  • $\begingroup$ They are randomly initialized, then optimized with the skip-gram algorithm. $\endgroup$
    – Cedias
    Commented May 21, 2016 at 9:55
  • $\begingroup$ Thanks for the explanation, that makes sense. I'll have another look at the paper. Feel free to put this into a short answer. $\endgroup$
    – danijar
    Commented May 21, 2016 at 10:08

1 Answer 1

Reset to default
2
$\begingroup$

It seems you have misunderstood how W2V algorithms work. Both W2V algorithms (Skip-Gram, Continuous BoW) use dense vectors initialized randomly which are optimized afterwards.

For skip-gram, because of this dense representation, context words are predicted in a sequence. (Christopher Moody does a great job explaining Skip-Gram here)

If you're familiar with C language you should read the original code here

Besides, Tensorflow documentation also provides a nice explanation of these models.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.