Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am currently trying to understand how the Word2Vec neural network works, but do not understand why we choose to take the weight vectors between the input and hidden layer as our word embedding vectors.
If we are using both sets of weights(input-hidden and hidden-output) to predict the context vectors from a word vector, why do we only the hidden-output weight vectors represent our word embeddings?
This is a really good question. I do not know the exact answer but have a feeling.
At the end of one epoch in the training the "input" vector (weights between input data and hidden layer) was the last updated along the process, since we are coming from the back of the model (using back-propagation) and the first weights visited are those between the hidden layer and the output, and so the last weights being updated are those between the input layer and the hidden layer (the so called "input vector") .
In addition the weights corresponding to the input vector are closest to the input data (the words in one-hot encoded format), and embedding vectors will be used as input to further NLP tasks.
There are additional answers here: Here
