# Confusion about CBOW and Skip-Gram models?

I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this:

1. We need to train models on words
2. So we encode words using vectors
3. One-hot encoding is not efficient representation, we deal with this issue with CBOW and Skip-Gram
4. CBOW is a model that allows you to predict the center word given the surrounding context
5. Skip-gram does the reverse, by allowing you to predict the context given a center word

...

But wait. The problem we had (at step 3) was that one-hot encoding does a poor job at representing words. How does predicting a word help with the encoding process?! It seems we completely forgot about what we were setting out to do.

I believe you're missing an essential point in both objectives in that we are trying to learn a distribute drepresentation $$\in \Re^d$$ as opposed to using the one hot encodings.

Consider the example of having a vocabulary of size $$|V|$$, then we can represent each word using a one hot encoding $$\in [0, 1]^{|V|}$$.

The skip-gram model will work as follows given a set of parameters $$E \in \Re^{d \times |V|}$$, and $$W \in \Re^{d \times |V|}$$:

1. Generate the one hot input vector $$x \in [0, 1]^{|V|}$$ for the center word
2. Obtain the embedding for the word vector $$e = Ex \in \Re^d$$
3. We then generate scores $$z = W^Te \in \Re^{|V|}$$ for the surrounding words

We can then normalize the scores using a softmax function $$y = \text{softmax}(z)$$, where $$y_i$$ are the probabilities of observing each context word. Here, the objective is to learn the parameters $$E$$ & $$W$$ so that our probabilities $$y$$ match the true one hot encodings of the context words given the center word.

I gave the example of skip-gram above, but CBOW is essentially the same but reversed. For more details on the objective function you could refer to , or for a visual explanation of the whole process .

• Hi. Thanks for the detailed explanation. So it seems that we first encode each word as a one-hot vector, then we obtain an embedding of this word, by adjusting the $E, W$ matrices so that the output $y$ match the one-hot encoding of the context words. So is our goal to generate embedding for all words in the vocabulary in this way? What do we do with these embedding after we have gotten them? Jun 22, 2020 at 18:15
• Also a problem seems to be that the embedding changes if you learn it using one set of context versus another set of context. Jun 22, 2020 at 18:17

The explanation of step 3 seems oversimplified, to extend it a bit:
> one-hot encoding is not efficient representation,
> so we want to use dense vector encoding instead,
> but unlike one-hot encoding it's not clear how should we represent discrete words as continuous vectors,
> we deal with this issue with CBOW and Skip-Gram.

First, "efficient representation" is a very vague term that you should avoid. For instance, one-hot encoding is way more efficient in terms of memory. It takes way, way more memory to encode 300 real-values than a single integer from 1 to $$|V|$$.