Confusion about CBOW and Skip-Gram models?

Question

I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this:

1. We need to train models on words
2. So we encode words using vectors
3. One-hot encoding is not efficient representation, we deal with this issue with CBOW and Skip-Gram
4. CBOW is a model that allows you to predict the center word given the surrounding context
5. Skip-gram does the reverse, by allowing you to predict the context given a center word

...

But wait. The problem we had (at step 3) was that one-hot encoding does a poor job at representing words. How does predicting a word help with the encoding process?! It seems we completely forgot about what we were setting out to do.

Answer 1

I believe you're missing an essential point in both objectives in that we are trying to learn a distribute drepresentation $\in \Re^d$ as opposed to using the one hot encodings.

Consider the example of having a vocabulary of size $|V|$, then we can represent each word using a one hot encoding $\in [0, 1]^{|V|}$.

The skip-gram model will work as follows given a set of parameters $E \in \Re^{d \times |V|}$, and $W \in \Re^{d \times |V|}$:

1. Generate the one hot input vector $x \in [0, 1]^{|V|}$ for the center word
2. Obtain the embedding for the word vector $e = Ex \in \Re^d$
3. We then generate scores $z = W^Te \in \Re^{|V|}$ for the surrounding words

We can then normalize the scores using a softmax function $y = \text{softmax}(z)$, where $y_i$ are the probabilities of observing each context word. Here, the objective is to learn the parameters $E$ & $W$ so that our probabilities $y$ match the true one hot encodings of the context words given the center word.

I gave the example of skip-gram above, but CBOW is essentially the same but reversed. For more details on the objective function you could refer to [1], or for a visual explanation of the whole process [2].

[1] https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

[2] http://jalammar.github.io/illustrated-word2vec/ [2]

Answer 2

The explanation of step 3 seems oversimplified, to extend it a bit:
> one-hot encoding is not efficient representation,
> so we want to use dense vector encoding instead,
> but unlike one-hot encoding it's not clear how should we represent discrete words as continuous vectors,
> we deal with this issue with CBOW and Skip-Gram.

Hope that answers your question.

Answer 3

First, "efficient representation" is a very vague term that you should avoid. For instance, one-hot encoding is way more efficient in terms of memory. It takes way, way more memory to encode 300 real-values than a single integer from 1 to $|V|$.

On the other hand, one-hot representation of words are not efficient statistically, because words are distributed according to heavy tailed distributions. That is, if you take a random word in your corpus, it will likely be a rare words. Conversely, frequent words account for a lot of the total number of occurences.

The consequence is clear on the following task. You want to classify text. If you use logistic regression on the sum of one-hot vectors (bag-of-word representation), the parameters associated with rare words will be fit based on very little data and the classifier might be unreliable. On the other hand, if you use the sum of word embeddings (bag-of-vector) there is no component that is estimated with little data so it should be more reliable.