7
$\begingroup$

In Word2Vec algorithm, two weight matrices are learnt :
W : Input-hidden layer matrix
W': Hidden-output layer matrix

For reference, CBOW model architecture:
CBOW Word2Vec Model architecture

  1. Why is W chosen to represent the word vectors and not W' ? They both seem to encode the same information.

  2. What is the interpretation of the W' matrix? Just like W represents word embeddings.

$\endgroup$
1
  • $\begingroup$ They both capture the word semantics. Not only W, sometimes W' is also used as word vectors. Even in somecases (W+W')/2 has also been used and better results in that particular task have been obtained. Another thing to notice is that no activation function is used after the hidden layer, so the transformation from input to output is W[i]*W'^T for any activated word i in input. So for every word vector you are trying to learn the words that mostly occurs in its vicinity(context-window). $\endgroup$
    – bytestorm
    Commented Apr 23, 2018 at 5:11

1 Answer 1

7
$\begingroup$

They both capture the word semantics. Not only W, sometimes W' is also used as word vectors. Even in somecases (W+W')/2 has also been used and better results in that particular task have been obtained.

Another thing to notice is that no activation function is used after the hidden layer, so the transformation from input to output is W[i]*W'^T for any activated word i in input. So for every word vector you are trying to learn the words that mostly occurs in its vicinity(context-window).

You can think of the two linear transformation as,

  • Semantics encoder from n-hot vector: Word-list to semantics
  • Semantics decoder which outputs a probability vector: Semantics to probability distribution over words.

Formally, vectors in W and W' are called input and output word vector representations, respectively.

$\endgroup$
2
  • 1
    $\begingroup$ thanks. It would be great if you can share some reference links of works where W' is being used for word vectors. $\endgroup$
    – Journ
    Commented Apr 24, 2018 at 17:35
  • $\begingroup$ Here is some interesting paper on combining the target and context embeddings. It also includes references to papers taking the average as mentioned here. $\endgroup$
    – thijsvdp
    Commented Oct 13, 2022 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.