2
$\begingroup$

I'm working on developing a better understanding of word embeddings, and am struggling a bit with understanding where pre-trained word embeddings come from. For instance, let's take Stanford's Wikipedia GLove pretrained embeddings. If you open up the unzipped text files, you'll see something like this:

the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581

with 0.25616 0.43694 -0.11889 0.20345 0.41959 0.85863 -0.60344 -0.31835 -0.6718 0.003984 -0.075159 0.11043 -0.73534 0.27436 0.054015 -0.23828 -0.13767 0.011573 -0.46623 -0.55233 0.083317 0.55938 0.51903 -0.27065 -0.28211 -1.3918 0.17498 0.26586 0.061449 -0.273 3.9032 0.38169 -0.056009 -0.004425 0.24033 0.30675 -0.12638 0.33436 0.075485 -0.036218 0.13691 0.37762 -0.12159 -0.13808 0.19505 0.22793 -0.17304 -0.07573 -0.25868 -0.39339

Essentially, I think of it as a dictionary with words as keys, and a vector embedding as the values.

Clearly, in this case, the length of the dictionary's keys must be V (the vocabulary size). The length of each vector embedding is D- for these embeddings, it looks like they are either 100, 200, 300.

I understand that these are essentially the resulting weight matrices after training the neural network architecture for some number of iterations:

enter image description here

However, I'm not clear which part of the weight matrices the word embeddings are actually coming from. Both are of shape ** N X V ** (although one is transposed): enter image description here

In a Pluralsight course I'm taking called Sentiment Analysis with Recurrent Neural Networks in TensorFlow by Janani Ravi, she says that for CBOW (continuous bag of words), it is B.

In traditional CS lectures on this topic (such as this one from Stanford), the lecturer annotates that "we must learn these weights", but it's not clear which set of weights is used: enter image description here

If the answer is A (or B), then my follow-up question is what information does the other weight matrix provide?

$\endgroup$
  • $\begingroup$ Since words are encoded as one-hot vectors, each column of the $\textbf{left}$ weight matrix $W$ corresponds to a word-embedding (if we assume the form $h = Wx)$. $\endgroup$ – user18764 Mar 14 '18 at 18:06
  • $\begingroup$ @user18764 so then the word embeddings are coming from the input to hidden weights? $\endgroup$ – Yu Chen Mar 14 '18 at 19:01
  • $\begingroup$ Yes, the embeddings are encoded in the transformation from $\textbf{x}$ to $\textbf{h}$. $\endgroup$ – user18764 Mar 14 '18 at 20:14
  • $\begingroup$ @user18764 Thanks. Then is there any information retained or use case for the weight matrix from hidden to output? Or is that entirely discarded? $\endgroup$ – Yu Chen Mar 14 '18 at 20:17
  • 1
    $\begingroup$ The second weight matrix gives information about how the embedded word $h$ relates to it's context vector $y$. The context vector is essentially the prediction of the distribution of words in the neighborhood of $x$. $\endgroup$ – user18764 Mar 15 '18 at 12:13
2
$\begingroup$

In comments, @user18764 writes:

Since words are encoded as one-hot vectors, each column of the left weight matrix W corresponds to a word-embedding (if we assume the form h=Wx). the embeddings are encoded in the transformation from x to h. The second weight matrix gives information about how the embedded word h relates to it's context vector y. The context vector is essentially the prediction of the distribution of words in the neighborhood of x.


I've copied this comment as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.

Please review

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.