0
$\begingroup$

I'm watching the following video on word2vec from University of Waterloo: https://www.youtube.com/watch?v=GMCwS7tS5ZM&t=962s

The update function for word my word embedding vector is:

$v'_w = v_w - v_c(1 - P(w|c))$, where

$v'_w$ = new word vector (embedding) of predicted word

$v_w$ = old word vector of predicted word

$v_c$ = old word vector of context word

$P(w|c)$ = "how well can I predict word given context"

The problem I'm having is that I don't see intuitively how this actually works. For example, if I initialize my $v$s to zero, I'll never actually make progress towards an accurate set of word embeddings, because the vectors will never move away from zero.

Question

What part of this am I missing? I feel like I'm not seeing the entire picture.

$\endgroup$
0
$\begingroup$

I think here intuition is easier then derivation. If the context word, $c$ can describe well target word, $w$, then $P(w|c)$ will be closer to $1$ compared to $0$. And, we don't subtract the context from our word. If it is close to $0$, i.e. the context word cannot describe the target very well, subtract its context value from the current target. Initialization is another issue. A good way is initializing your weights to small, zero mean random values. Don't ever initialize all to zero. And, specifically in your case, it's a local minimum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.