Understanding word2vec backpropagation

Question

I'm watching the following video on word2vec from University of Waterloo: https://www.youtube.com/watch?v=GMCwS7tS5ZM&t=962s

The update function for word my word embedding vector is:

$v'_w = v_w - v_c(1 - P(w|c))$, where

$v'_w$ = new word vector (embedding) of predicted word

$v_w$ = old word vector of predicted word

$v_c$ = old word vector of context word

$P(w|c)$ = "how well can I predict word given context"

The problem I'm having is that I don't see intuitively how this actually works. For example, if I initialize my $v$s to zero, I'll never actually make progress towards an accurate set of word embeddings, because the vectors will never move away from zero.

Question

What part of this am I missing? I feel like I'm not seeing the entire picture.

Answer 1

I think here intuition is easier then derivation. If the context word, $c$ can describe well target word, $w$, then $P(w|c)$ will be closer to $1$ compared to $0$. And, we don't subtract the context from our word. If it is close to $0$, i.e. the context word cannot describe the target very well, subtract its context value from the current target. Initialization is another issue. A good way is initializing your weights to small, zero mean random values. Don't ever initialize all to zero. And, specifically in your case, it's a local minimum.