# Which part of the hidden layer architecture do pretrained word embeddings come from?

I'm working on developing a better understanding of word embeddings, and am struggling a bit with understanding where pre-trained word embeddings come from. For instance, let's take Stanford's Wikipedia GLove pretrained embeddings. If you open up the unzipped text files, you'll see something like this:

the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581

with 0.25616 0.43694 -0.11889 0.20345 0.41959 0.85863 -0.60344 -0.31835 -0.6718 0.003984 -0.075159 0.11043 -0.73534 0.27436 0.054015 -0.23828 -0.13767 0.011573 -0.46623 -0.55233 0.083317 0.55938 0.51903 -0.27065 -0.28211 -1.3918 0.17498 0.26586 0.061449 -0.273 3.9032 0.38169 -0.056009 -0.004425 0.24033 0.30675 -0.12638 0.33436 0.075485 -0.036218 0.13691 0.37762 -0.12159 -0.13808 0.19505 0.22793 -0.17304 -0.07573 -0.25868 -0.39339

Essentially, I think of it as a dictionary with words as keys, and a vector embedding as the values.

Clearly, in this case, the length of the dictionary's keys must be V (the vocabulary size). The length of each vector embedding is D- for these embeddings, it looks like they are either 100, 200, 300.

I understand that these are essentially the resulting weight matrices after training the neural network architecture for some number of iterations:

However, I'm not clear which part of the weight matrices the word embeddings are actually coming from. Both are of shape ** N X V ** (although one is transposed):

In a Pluralsight course I'm taking called Sentiment Analysis with Recurrent Neural Networks in TensorFlow by Janani Ravi, she says that for CBOW (continuous bag of words), it is B.

In traditional CS lectures on this topic (such as this one from Stanford), the lecturer annotates that "we must learn these weights", but it's not clear which set of weights is used:

If the answer is A (or B), then my follow-up question is what information does the other weight matrix provide?

• Since words are encoded as one-hot vectors, each column of the $\textbf{left}$ weight matrix $W$ corresponds to a word-embedding (if we assume the form $h = Wx)$. Mar 14, 2018 at 18:06
• @user18764 so then the word embeddings are coming from the input to hidden weights? Mar 14, 2018 at 19:01
• Yes, the embeddings are encoded in the transformation from $\textbf{x}$ to $\textbf{h}$. Mar 14, 2018 at 20:14
• @user18764 Thanks. Then is there any information retained or use case for the weight matrix from hidden to output? Or is that entirely discarded? Mar 14, 2018 at 20:17
• The second weight matrix gives information about how the embedded word $h$ relates to it's context vector $y$. The context vector is essentially the prediction of the distribution of words in the neighborhood of $x$. Mar 15, 2018 at 12:13