5
$\begingroup$

As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at: https://www.youtube.com/watch?v=ERibwqs9p38 (Highly appreciate Stanford university and prof Manning sharing the video)

Two questions I struggled:

  1. At the place where the center word vector multiplies the context word matrix to generate three UoT*Vc vectors. Given the same center word vector and the same context matrix, why it came out three different vectors?

  2. The output of the softmax function is a list of probabilities. A list of probabilities that showing me the likelihood of each word in the vocabulary being the context word of the given center word. Wouldn't just having one such vector be good enough? Why the graphs shows me three (different) such vectors for each of the context word?

enter image description here

I found a similar (may not be exact) question from the following link, but still didn't feel fully answered.

Improve this question
$\endgroup$
1
  • $\begingroup$ In my opinion the different distributions are for different places in the context window , since for each place there is a different truth value . $\endgroup$
    – Dhruv
    Dec 24, 2018 at 18:32

1 Answer 1

Reset to default
5
$\begingroup$

The skipgram model produces one softmax output for all the context words. The drawing from the lecture is incorrect.

Improve this answer
$\endgroup$
4
  • $\begingroup$ For a given center word vector, it multiplies with the same context word matrix, it should get the same vector output 3 times. If you listen to the youtube video, the professor emphasized that it is the same context matrix for all three context words. But the graph is showing three different UoTVc ones. That's where I do not understand. Can you please explain why there would be three UoTVc given dot products of same input vector and matrix. Thanks. $\endgroup$
    – MeiNan Zhu
    Apr 20, 2018 at 21:07
  • $\begingroup$ @MeiNanZhu it seems i was also misunderstanding the model. I fixed my answer. $\endgroup$
    – shimao
    Apr 20, 2018 at 21:41
  • $\begingroup$ @shimao, what would a correct network for the architecture be then? $\endgroup$
    – edamondo
    Aug 27 at 15:50
  • $\begingroup$ @shimao, oh I think I found a correct one here $\endgroup$
    – edamondo
    Aug 27 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.