Which part of the hidden layer architecture do pretrained word embeddings come from?

Question

I'm working on developing a better understanding of word embeddings, and am struggling a bit with understanding where pre-trained word embeddings come from. For instance, let's take Stanford's Wikipedia GLove pretrained embeddings. If you open up the unzipped text files, you'll see something like this:

the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581

with 0.25616 0.43694 -0.11889 0.20345 0.41959 0.85863 -0.60344 -0.31835 -0.6718 0.003984 -0.075159 0.11043 -0.73534 0.27436 0.054015 -0.23828 -0.13767 0.011573 -0.46623 -0.55233 0.083317 0.55938 0.51903 -0.27065 -0.28211 -1.3918 0.17498 0.26586 0.061449 -0.273 3.9032 0.38169 -0.056009 -0.004425 0.24033 0.30675 -0.12638 0.33436 0.075485 -0.036218 0.13691 0.37762 -0.12159 -0.13808 0.19505 0.22793 -0.17304 -0.07573 -0.25868 -0.39339

Essentially, I think of it as a dictionary with words as keys, and a vector embedding as the values.

Clearly, in this case, the length of the dictionary's keys must be V (the vocabulary size). The length of each vector embedding is D- for these embeddings, it looks like they are either 100, 200, 300.

I understand that these are essentially the resulting weight matrices after training the neural network architecture for some number of iterations:

However, I'm not clear which part of the weight matrices the word embeddings are actually coming from. Both are of shape ** N X V ** (although one is transposed):

In a Pluralsight course I'm taking called Sentiment Analysis with Recurrent Neural Networks in TensorFlow by Janani Ravi, she says that for CBOW (continuous bag of words), it is B.

In traditional CS lectures on this topic (such as this one from Stanford), the lecturer annotates that "we must learn these weights", but it's not clear which set of weights is used:

If the answer is A (or B), then my follow-up question is what information does the other weight matrix provide?

Answer 1

In comments, @user18764 writes:

Since words are encoded as one-hot vectors, each column of the left weight matrix W corresponds to a word-embedding (if we assume the form h=Wx). the embeddings are encoded in the transformation from x to h. The second weight matrix gives information about how the embedded word h relates to it's context vector y. The context vector is essentially the prediction of the distribution of words in the neighborhood of x.

---

I've copied this comment as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.

Please review

• Are we seeing a dramatic drop in answers per question?

• Comments that are actually answers