I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this:

  1. We need to train models on words
  2. So we encode words using vectors
  3. One-hot encoding is not efficient representation, we deal with this issue with CBOW and Skip-Gram
  4. CBOW is a model that allows you to predict the center word given the surrounding context
  5. Skip-gram does the reverse, by allowing you to predict the context given a center word


But wait. The problem we had (at step 3) was that one-hot encoding does a poor job at representing words. How does predicting a word help with the encoding process?! It seems we completely forgot about what we were setting out to do.


I believe you're missing an essential point in both objectives in that we are trying to learn a distribute drepresentation $\in \Re^d$ as opposed to using the one hot encodings.

Consider the example of having a vocabulary of size $|V|$, then we can represent each word using a one hot encoding $\in [0, 1]^{|V|}$.

The skip-gram model will work as follows given a set of parameters $E \in \Re^{d \times |V|}$, and $W \in \Re^{d \times |V|}$:

  1. Generate the one hot input vector $x \in [0, 1]^{|V|}$ for the center word
  2. Obtain the embedding for the word vector $e = Ex \in \Re^d$
  3. We then generate scores $z = W^Te \in \Re^{|V|}$ for the surrounding words

We can then normalize the scores using a softmax function $y = \text{softmax}(z)$, where $y_i$ are the probabilities of observing each context word. Here, the objective is to learn the parameters $E$ & $W$ so that our probabilities $y$ match the true one hot encodings of the context words given the center word.

I gave the example of skip-gram above, but CBOW is essentially the same but reversed. For more details on the objective function you could refer to [1], or for a visual explanation of the whole process [2].

[1] https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

[2] http://jalammar.github.io/illustrated-word2vec/ [2]

  • $\begingroup$ Hi. Thanks for the detailed explanation. So it seems that we first encode each word as a one-hot vector, then we obtain an embedding of this word, by adjusting the $E, W$ matrices so that the output $y$ match the one-hot encoding of the context words. So is our goal to generate embedding for all words in the vocabulary in this way? What do we do with these embedding after we have gotten them? $\endgroup$ – Rodrigo Amarante Jun 22 '20 at 18:15
  • $\begingroup$ Also a problem seems to be that the embedding changes if you learn it using one set of context versus another set of context. $\endgroup$ – Rodrigo Amarante Jun 22 '20 at 18:17

The explanation of step 3 seems oversimplified, to extend it a bit:
> one-hot encoding is not efficient representation,
> so we want to use dense vector encoding instead,
> but unlike one-hot encoding it's not clear how should we represent discrete words as continuous vectors,
> we deal with this issue with CBOW and Skip-Gram.

Hope that answers your question.

  • $\begingroup$ What do we do with this dense representation afterwards? $\endgroup$ – Rodrigo Amarante Jun 22 '20 at 18:17
  • $\begingroup$ @Carlos-theMongoose-Danger this dense representation is just another way of encoding text, whatever can be done with one-hot representation you can try using the dense representation $\endgroup$ – dontloo Jun 22 '20 at 20:28

First, "efficient representation" is a very vague term that you should avoid. For instance, one-hot encoding is way more efficient in terms of memory. It takes way, way more memory to encode 300 real-values than a single integer from 1 to $|V|$.

On the other hand, one-hot representation of words are not efficient statistically, because words are distributed according to heavy tailed distributions. That is, if you take a random word in your corpus, it will likely be a rare words. Conversely, frequent words account for a lot of the total number of occurences.

The consequence is clear on the following task. You want to classify text. If you use logistic regression on the sum of one-hot vectors (bag-of-word representation), the parameters associated with rare words will be fit based on very little data and the classifier might be unreliable. On the other hand, if you use the sum of word embeddings (bag-of-vector) there is no component that is estimated with little data so it should be more reliable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.