I wonder why skip-gram is better for infrequent words than CBOW in word2vec. I have read the claim on https://code.google.com/p/word2vec/.
Here is my oversimplified and rather naive understanding of the difference:
As we know, CBOW is learning to predict the word by the context. Or maximize the probability of the target word by looking at the context. And this happens to be a problem for rare words. For example, given the context yesterday was really [...] day
CBOW model will tell you that most probably the word is beautiful
or nice
. Words like delightful
will get much less attention of the model, because it is designed to predict the most probable word. Rare words will be smoothed over a lot of examples with more frequent words.
On the other hand, the skip-gram is designed to predict the context. Given the word delightful
it must understand it and tell us, that there is huge probability, the context is yesterday was really [...] day
, or some other relevant context. With skip-gram the word delightful
will not try to compete with word beautiful
but instead, delightful+context
pairs will be treated as new observations. Because of this, skip-gram will need more data so it will learn to understand even rare words.
In CBOW the vectors from the context words are averaged before predicting the center word. In skip-gram there is no averaging of embedding vectors. It seems like the model can learn better representations for the rare words when their vectors are not averaged with the other context words in the process of making the predictions.
I have just come across a paper that shows the opposite: that CBOW is better for infrequent words than skip-gram https://arxiv.org/abs/1609.08293. I wonder what are the sources of the stated claim on https://code.google.com/p/word2vec/.
-
$\begingroup$ I believe Mikolov wrote that toolkit himself. Interestingly, in his paper: papers.nips.cc/paper/… he states "We show that subsampling of frequent words during training results in a significant speedup (around 2x - 10x), and improves accuracy of the representations of less frequent words." so its skip-gram with the subsampling extension. $\endgroup$ – Kevin Jan 21 '19 at 3:56