I am trying to reimplement skip-gram and CBOW. I think I understand the architecture well, but I am confused on how to input pairs are exactly generated.
For skip-gram, based on McCormick's post, a sentence like "I like soup very much" would become five training examples, assuming "soup" is the center word, and the window size is 2:
soup, I
soup, like
soup, very
soup, much
Then, when I am reading an actual implementation from stanford cs20i. It introduced randomness, basically, instead of using the specified window size ($w$), it chose a value between 1 and $w$. Effectively, I think it downsamples the training data, but can anyone explain why this is necessary? I thought the biggest selling points of word2vec is to leverage as much data as possible with a computationally efficient model, why downsample then.
For CBOW, I am confused reading different sources. At least one version seems to be averaging the input words within the context and then predict the center word. e.g. assuming using the same sentence and window size as above, there is only one training data point corresponding to the center word
(I + like + very + much) / 4, soup
I have two questions regarding CBOW:
First, is this the way done in the word2vec paper, Efficient Estimation of Word Representations in Vector Space? I am somewhat confused reading it.
Second, can we just do the exact opposite to the skip-gram way of generating word pairs? Then, we would get
I, soup
like, soup
very, soup
much, soup
which are more training examples, aren't they?
As I understand it, all input/output word pairs are represented in one-hot vectors.