0
$\begingroup$

Sorry for the beginner level question, but I am fairly new to the NLP world and am trying to better understand how word2vec is able to create useful word embeddings.

I'm looking for an intuitive explanation as to why, when creating word embeddings with a word2vec algorithm, the weight matrix should contain useful information relating to the word embeddings. As I understand it, when using either a CBOW or skip-gram model, we input one-hot vectors into a shallow neural network, and train the network to predict either the missing word, or the context words (depending on whether we're using a CBOW or skip-gram approach). Then we take the columns of the learned weight matrix to be the vector representations of the words in our vocabulary.

What I can't understand/internalize is exactly why the weight matrix generated by the network should actually have columns that represent useful word embeddings of each word (i.e. why should the network necessarily contain columns such that words with similar meanings are near each other in the embedding space?). It seems like you could imagine the network learning to create a weight matrix that does the specific task (predicting the missing word/predicting the context words) without the matrix having anything to do with word embeddings.

Perhaps I'm just missing something silly here, but if somebody could give an explanation to this, it would help me a lot!

$\endgroup$

1 Answer 1

0
$\begingroup$

Multiplying the one-hot vector with the weight matrix actually means selecting one column from the weight matrix. This is the reason why the words are one-to-one associated with the columns of the weight matrix. In practice, there are no one-hot vectors and the embedding lookup is implemented directly as retrieving the corresponding vector from the matrix.

The vector from the weight matrix is trained in such a way that you predict what words typically appear around the particular word which says how to word is used which is a proxy for its meaning (or in some theories, such as in Wittgenstein's Philosophical Investigations, the use is actually the meaning). The context actually predetermines what word can be there: both its grammatical features and its possible meaning.

$\endgroup$
2
  • $\begingroup$ Thanks for the reply! So the intuition is the following: the column in the weight matrix for a given target word has values that "connect" it to words that appear near it. Other words with similar context to the target word should have similar "connections", thus words that show up in similar contexts end up in similar locations in the embedding space. Is this more or less correct? $\endgroup$ Oct 4, 2022 at 16:21
  • $\begingroup$ Yes, this sounds right. $\endgroup$
    – Jindřich
    Oct 5, 2022 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.