Why is skip-gram better for infrequent words than CBOW?

Question

I wonder why skip-gram is better for infrequent words than CBOW in word2vec. I have read the claim on https://code.google.com/p/word2vec/.

Answer 1

Here is my oversimplified and rather naive understanding of the difference:

As we know, CBOW is learning to predict the word by the context. Or maximize the probability of the target word by looking at the context. And this happens to be a problem for rare words. For example, given the context yesterday was really [...] day CBOW model will tell you that most probably the word is beautiful or nice. Words like delightful will get much less attention of the model, because it is designed to predict the most probable word. Rare words will be smoothed over a lot of examples with more frequent words.

On the other hand, the skip-gram is designed to predict the context. Given the word delightful it must understand it and tell us, that there is huge probability, the context is yesterday was really [...] day, or some other relevant context. With skip-gram the word delightful will not try to compete with word beautiful but instead, delightful+context pairs will be treated as new observations. Because of this, skip-gram will need more data so it will learn to understand even rare words.

Answer 2

In CBOW the vectors from the context words are averaged before predicting the center word. In skip-gram there is no averaging of embedding vectors. It seems like the model can learn better representations for the rare words when their vectors are not averaged with the other context words in the process of making the predictions.

Answer 3

I have just come across a paper that shows the opposite: that CBOW is better for infrequent words than skip-gram https://arxiv.org/abs/1609.08293. I wonder what are the sources of the stated claim on https://code.google.com/p/word2vec/.