3
$\begingroup$

I'm following Udacity Deep learning course and the instructor briefly explained about the "Sampled Softmax" used in word2vec.

In tensorflow word2vec_basic.py, the implementation is like below.

for j in range(num_skips):
  while target in targets_to_avoid:
    target = random.randint(0, span - 1) # randomly choose a word from the context of target word
  targets_to_avoid.append(target)

I'm having hard time understanding why randomly remove some words would work well even if it reduces the computaion. Can someone shed some light on it?

$\endgroup$
2
$\begingroup$

I'm not fully across this implementation, but I think I understand what you're asking.

If we randomly skip words in the context window, we're still going to see all of them, on average.

We loop through the dataset multiple times so we're going to train on the same example many times. Each time we'll probably see a different subset of the words in the window, but on average we'll see all of them.

Hope that's clear / I understood what you're asking.

$\endgroup$
  • $\begingroup$ Is that a kind of dropout? en.wikipedia.org/wiki/Dropout_(neural_networks) $\endgroup$ – hamster on wheels Aug 4 '17 at 20:54
  • $\begingroup$ Not really. This won't make the network any more (or less) robust as per dropout. It's closer in principle to subsampling an over-represented class when dealing with class-imbalanced data. $\endgroup$ – nlml Aug 4 '17 at 21:01
  • $\begingroup$ Yup, your answer is what I was looking for. Thanks!! $\endgroup$ – Aaron Aug 14 '17 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.