1
$\begingroup$

I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values for words are assigned?....also, once assigned, how/why are the vectors then adjusted in the algorithm?

$\endgroup$

1 Answer 1

1
$\begingroup$

Word2vec is not any more exciting or complicated than a simple, single-layer neural network. If you’re familiar with those, a lot of the same information applies here.

The vectors for each word are randomly initialized.

As the model is fitted, the optimizer improves the utility of the vectors; it makes them more capable of predicting the word from context (CBOW) or context from word (skip-gram). It considers each word-in-context one by one. (If you’re familiar with batching, this process can also be batched.) This is done by computing a loss function in the forward pass (“how far off is my ability to predict the right word?”), computing gradients of this loss with respect to the vectors in the backward pass, and updating the weights accordingly.

$\endgroup$
2
  • $\begingroup$ Thank you Arya...I'm getting a better undestanding now. What I'm wondering now is if/how the output values to be predicted are changed? so, you assign vector values randomly and optimise them to predict the output values (which are presumably randomly assigned too). Are the output values to be predicted changed in subsequent iterations too? $\endgroup$
    – osckt
    Jul 10, 2023 at 13:01
  • $\begingroup$ Yes, that's correct. All are jointly optimized. $\endgroup$ Jul 10, 2023 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.