3
$\begingroup$

Based on this post on quora.com and other sources I got the impression that each word in the word2vec representation is represented by a vector containing e.g. 500 dimensions. However, when looking into the code on sentiment analysis from the tutorial at deeplearning.net, I've found that each sentence is simply captured in a vector with dimensions of ~40-80. One such example is below:

x = [17, 25, 769, 83, 3, 14, 80, 62, 3221, 5, 928, 3, 1782, 6, 1, 1, 771, 24, 3350, 7, 1112, 228, 5, 3978, 4, 17, 25, 1212, 80, 6, 189, 7, 62, 1293, 5, 514, 4, 2, 131, 10, 1146, 480, 59, 413, 213, 117, 3, 14, 824, 69, 611, 2, 239, 73, 222, 72, 2338, 2, 67, 147, 4, 15, 1164, 123, 17, 10, 6, 100, 111, 23, 45, 228, 25, 4427, 8, 1131, 73, 31, 4]
y = [1]

With a sentence consisting of many words, it is clear that this vector cannot represent each word using 200 or 500 dimensions. Instead it seems that each word is approxmiated by a single scalar value. How can this be?

$\endgroup$
  • $\begingroup$ where did you see the 40-80 figure? The only data on size I could find is in lstm.py. Function def train_lstm dim_proj=128, # word embeding dimension and LSTM number of hidden units. That is, they use 128 dimensions vector for each word $\endgroup$ – Jacques Wainer Aug 20 '15 at 15:44
  • $\begingroup$ If you look at the values for x and y at roughly line 565 in lstm.py in a debugger (or print them), you'll see that they are lists with each element being similar to the x and y in my original post. Each element in x is a single sentence captured in a single vector with ~40-80 values. $\endgroup$ – pir Aug 20 '15 at 17:08
2
$\begingroup$

Each word in a given sentence is converted into an index on an array that contains a dictionary of all words in your corpus. This is your variable length x vector and is separate from the word2vec embeddings.

This index is also used to lookup the associated fixed length word2vec vectors in a lookup table during training and prediction.

This RNN deeplearning tutorial explains it better with code samples.

In torch this word index lookup is handled by the Lookup table layer.

$\endgroup$
  • 1
    $\begingroup$ Thanks! Can you help me find where in the accompanying scripts for deeplearning.net/tutorial/lstm.html I set the number of dimensions for the word2vec representation? $\endgroup$ – pir Sep 3 '15 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.