Why would the sampled softmax work? [word2vec]

Question

I'm following Udacity Deep learning course and the instructor briefly explained about the "Sampled Softmax" used in word2vec.

In tensorflow word2vec_basic.py, the implementation is like below.

for j in range(num_skips):
  while target in targets_to_avoid:
    target = random.randint(0, span - 1) # randomly choose a word from the context of target word
  targets_to_avoid.append(target)

I'm having hard time understanding why randomly remove some words would work well even if it reduces the computaion. Can someone shed some light on it?

Answer 1

I'm not fully across this implementation, but I think I understand what you're asking.

If we randomly skip words in the context window, we're still going to see all of them, on average.

We loop through the dataset multiple times so we're going to train on the same example many times. Each time we'll probably see a different subset of the words in the window, but on average we'll see all of them.

Hope that's clear / I understood what you're asking.