19
$\begingroup$

I wonder why skip-gram is better for infrequent words than CBOW in word2vec. I have read the claim on https://code.google.com/p/word2vec/.

$\endgroup$
14
$\begingroup$

In CBOW the vectors from the context words are averaged before predicting the center word. In skip-gram there is no averaging of embedding vectors. It seems like the model can learn better representations for the rare words when their vectors are not averaged with the other context words in the process of making the predictions.

$\endgroup$
13
$\begingroup$

Here is my oversimplified and rather naive understanding of the difference:

As we know, CBOW is learning to predict the word by the context. Or maximize the probability of the target word by looking at the context. And this happens to be a problem for rare words. For example, given the context yesterday was really [...] day CBOW model will tell you that most probably the word is beautiful or nice. Words like delightful will get much less attention of the model, because it is designed to predict the most probable word. Rare words will be smoothed over a lot of examples with more frequent words.

On the other hand, the skip-gram is designed to predict the context. Given the word delightful it must understand it and tell us, that there is huge probability, the context is yesterday was really [...] day, or some other relevant context. With skip-gram the word delightful will not try to compete with word beautiful but instead, delightful+context pairs will be treated as new observations. Because of this, skip-gram will need more data so it will learn to understand even rare words.

$\endgroup$
0
$\begingroup$

I have just come across a paper that shows the opposite: that CBOW is better for infrequent words than skip-gram https://arxiv.org/abs/1609.08293. I wonder what are the sources of the stated claim on https://code.google.com/p/word2vec/.

$\endgroup$
  • $\begingroup$ I believe Mikolov wrote that toolkit himself. Interestingly, in his paper: papers.nips.cc/paper/… he states "We show that subsampling of frequent words during training results in a significant speedup (around 2x - 10x), and improves accuracy of the representations of less frequent words." so its skip-gram with the subsampling extension. $\endgroup$ – Kevin Jan 21 '19 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.