Understanding Training Word Embeddings?

Question

I am new to Natural Language Processing. I am trying to understand how word embeddings are created.

1. When we are training Neural Networks, it is usuallly the weights of the neural network that are changed. So for when we are training in a mechanism like auto-encoders (which according to me is: given an input sequence "stackoverflow community is great" will try to predict "stackoverflow community is great". To do this the weights of the neural network is trained.

2. Now if only the weights of the neural network is trained, how exactly do we obtain the word embeddings (that is how do we get vectors for words, so that we can do things like King-Queen+Woman=Man or something similar. How exactly are we getting the word vectors if we are only changing neural network weights and not the input vector components).

Some resources I have looked at:

1. Generating Code Embeddings Part 1

   Generating Code Embeddings Part 2

   Generating Code Embeddings Part 3

2. Reddit Anything2vec

3. Word2vec

4. Generate your own embeddings

Answer 1

The word embeddings are weights in the model. They get learned just like any other. What you input to your model isn't a word embedding. Instead, you provide one-hot vectors for each word. These get multiplied by the matrix of embeddings to select the embedding for each word. That matrix is a parameter of your model.

Equivalently and more space-efficiently, you can provide a numerical index to select the row with the word's embedding, rather than the one-hot vector. The multiplication route says that we multiply the embedding matrix W by [0 1 0 0 0]; the indexing route says that we select the row W[2] instead.

Either way, all of the embeddings are stored in that matrix W which is a parameter of the model. There is one row for each word in the model's vocabulary.