Why should the weight matrix encode word embeddings in CBOW/skip-gram?

Question

Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings.

I'm looking for an intuitive explanation as to why, when creating word embeddings with a word2vec algorithm, the weight matrix should contain useful information relating to the word embeddings. As I understand it, when using either a CBOW or skip-gram model, we input one-hot vectors into a shallow neural network, and train the network to predict either the missing word, or the context words (depending on whether we're using a CBOW or skip-gram approach). Then we take the columns of the learned weight matrix to be the vector representations of the words in our vocabulary.

What I can't understand/internalize is exactly why the weight matrix generated by the network should actually have columns that represent useful word embeddings of each word (i.e. why should the network necessarily contain columns such that words with similar meanings are near each other in the embedding space?). It seems like you could imagine the network learning to create a weight matrix that does the specific task (predicting the missing word/predicting the context words) without the matrix having anything to do with word embeddings.

Perhaps I'm just missing something silly here, but if somebody could give an explanation to this, it would help me a lot!

Answer 1

Multiplying the one-hot vector with the weight matrix actually means selecting one column from the weight matrix. This is the reason why the words are one-to-one associated with the columns of the weight matrix. In practice, there are no one-hot vectors and the embedding lookup is implemented directly as retrieving the corresponding vector from the matrix.

The vector from the weight matrix is trained in such a way that you predict what words typically appear around the particular word which says how to word is used which is a proxy for its meaning (or in some theories, such as in Wittgenstein's Philosophical Investigations, the use is actually the meaning). The context actually predetermines what word can be there: both its grammatical features and its possible meaning.