I understand that in case of seq2seq learning for machine translation or question-answer systems one must encode words of sentences. Most approaches do this via word2vec representation where each word has a given long vector representation.

But I struggle to understand why this effort with word2vec (or any other similar method) is needed when one could represent the words simply with an ID scalar from a preconstructed dictionary (and indeed, Google's Neural Machine tutorial uses this simple way), so each sentence is turned to a 1D vector with ID integer values. What is the additional benefit of the vector representation?


Do you mean by "ID scalar" really one number, so 1: Dog, 2: Cat, or a one hot encoded vector, so {1,0} Dog, {0,1} Cat.

First Approach has the downside, that you already imply a n order/distance by construction, so for example if you encode: 1:Dog, 2:Tea, 3:Cat, you say that Dog and Tea are more simliar than Dog and Cat.

So the first improvement is the one hot encoding, so one dimension per word. Problem here is, that for a big vocabulary you get huge vectors, which is first of all harder to store, and second your performance suffers from the curse of dimension, and third now you dont have any distance anymore (e.q. Dog, Cat, and Tea have all the same distance)

If you do word embedding like word2vec, you reduce the dimension, since the target dimesnion is usually choosen way smaller than the vocabulary size, and in this newly constructed space Cat and Dog tend to be nearer than Cat and Tea (plus you get some surprising effects like word arithmetics, so if you take vectors for King - Men + Women = Queen )


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.