As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at: https://www.youtube.com/watch?v=ERibwqs9p38 (Highly appreciate Stanford university and prof Manning sharing the video)

Two questions I struggled:

  1. At the place where the center word vector multiplies the context word matrix to generate three UoT*Vc vectors. Given the same center word vector and the same context matrix, why it came out three different vectors?

  2. The output of the softmax function is a list of probabilities. A list of probabilities that showing me the likelihood of each word in the vocabulary being the context word of the given center word. Wouldn't just having one such vector be good enough? Why the graphs shows me three (different) such vectors for each of the context word?

enter image description here

I found a similar (may not be exact) question from the following link, but still didn't feel fully answered.

  • $\begingroup$ In my opinion the different distributions are for different places in the context window , since for each place there is a different truth value . $\endgroup$ – Dhruv Dec 24 '18 at 18:32

The skipgram model produces one softmax output for all the context words. The drawing from the lecture is incorrect.

  • $\begingroup$ For a given center word vector, it multiplies with the same context word matrix, it should get the same vector output 3 times. If you listen to the youtube video, the professor emphasized that it is the same context matrix for all three context words. But the graph is showing three different UoTVc ones. That's where I do not understand. Can you please explain why there would be three UoTVc given dot products of same input vector and matrix. Thanks. $\endgroup$ – MeiNan Zhu Apr 20 '18 at 21:07
  • $\begingroup$ @MeiNanZhu it seems i was also misunderstanding the model. I fixed my answer. $\endgroup$ – shimao Apr 20 '18 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.