I'm reading CNN for text classification. According to this link, inside data and preprocessing step it says:

  1. Load positive and negative sentences from the raw data files.
  2. Clean the text data using the same code as the original paper.
  3. Pad each sentence to the maximum sentence length, which turns out to be 59. We append special tokens to all other sentences to make them 59 words. Padding sentences to the same length is useful because it allows us to efficiently batch our data since each example in a batch must be of the same length.
  4. Build a vocabulary index and map each word to an integer between 0 and 18,765 (the vocabulary size). Each sentence becomes a vector of integers.

I read the paper on word2vec, all it says is dealing with one hot vector.

I'm not able to connect step 4 from above blog to paper.

  • 1
    $\begingroup$ Are they mentioning "one hot vector" in Google's word2vec paper? As stated in the first article you point to, they do not "use[] pre-trained word2vec vectors for our word embeddings". $\endgroup$ – tagoma Dec 20 '17 at 9:07
  • $\begingroup$ Yes, check in section 2.1 feedforward Neural Net Language Model. Yes, they are not using pre-trained embeddings. They will train from scratch $\endgroup$ – Bhaskar Dhariyal Dec 20 '17 at 9:26

Suppose there's an embedding matrix $M$, of size $V \times d$, where $V$ is the vocabulary size and $d$ is the embedding size. Suppose further an input sentence $(w_1, ..., w_k)$ that is encoded by indices $(i_1, ..., i_k)$, where $i_j \leq V$.

In theory, the following two approaches are equivalent:

  • Select the rows from the embedding matrix $M$ that correspond to indices $(i_1, ..., i_k)$.

  • Convert $(i_1, ..., i_k)$ to one-hot encoded matrix $O$, of size $k \times V$, and compute the dot-product $O \cdot M$, like on the picture below.


In both cases, the result is a matrix of embeddings for all words in the sentence: $k \times d$. But in programming, the first approach is much more efficient than the second one. Both in terms of computational complexity (select is a cheaper operation than dot-product) and memory (it uses O(k) memory for the input instead of O(k*V)). Especially when $V$ is large, tens or even hundreds of thousands. Remember that optimization is usually done in batches, not by one sentence at a time.

That's why there's tf.nn.embedding_lookup function in tensorflow that accepts sparse representation (the tensor of indices, not one-hot vectors), and no wonder the tutorial that you refer to uses it. Though word2vec paper talks about one-hot vectors, I believe that in code they are using indices as well, because Google's 1T vocabulary size is 13M!

So, in general, it's better to avoid one-hot representation when the number of classes is large, like natural language vocabulary.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.