4
$\begingroup$

I am currently trying to understand how the Word2Vec neural network works, but do not understand why we choose to take the weight vectors between the input and hidden layer as our word embedding vectors.

If we are using both sets of weights(input-hidden and hidden-output) to predict the context vectors from a word vector, why do we only the hidden-output weight vectors represent our word embeddings?

$\endgroup$
4
  • 3
    $\begingroup$ it's not clear what you're struggling with. if you put a few equations and point to the part that is confusing, it'll be easier to answer the question $\endgroup$
    – Aksakal
    Mar 19 '18 at 15:20
  • 1
    $\begingroup$ This is a really great question. I don't know any reference that has the answer but I suspect that it doesn't make much of a difference which one you use. $\endgroup$
    – Aaron
    Mar 24 '18 at 5:30
  • $\begingroup$ The weight matrix b/t the input and hidden layer is shared between all words, right? So it wouldn't be useful for learning distinct embeddings. $\endgroup$
    – shimao
    Mar 26 '18 at 10:44
  • $\begingroup$ It is a consistent mapping/representation of all words across your vocabulary. Think of them as abstract features. $\endgroup$
    – Edv Beq
    May 28 at 3:03
0
$\begingroup$

This is a really good question. I do not know the exact answer but have a feeling.

At the end of one epoch in the training the "input" vector (weights between input data and hidden layer) was the last updated along the process, since we are coming from the back of the model (using back-propagation) and the first weights visited are those between the hidden layer and the output, and so the last weights being updated are those between the input layer and the hidden layer (the so called "input vector") .

In addition the weights corresponding to the input vector are closest to the input data (the words in one-hot encoded format), and embedding vectors will be used as input to further NLP tasks.

There are additional answers here: Here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.