I was implementing word2vec in TensorFlow and found that Gradient Descent worked much better and faster than the AdamOptimizer. I was under the impression that Adam was the "smarter" option that almost always does better than GD. I used several starting learning rates for Adam, from 1.0 to 0.01, but none did nearly as well as GD with a learning rate of 1.0. Am I missing something about these optimizers or their application to word2vec in particular?


# Define the placeholders for input and output
center_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE], name='center_words')
target_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE, 1], name='target_words')

# Define weights. In word2vec, it's actually the weights that we care about
embed_matrix = tf.Variable(tf.random_uniform([VOCAB_SIZE, EMBED_SIZE], -1.0, 1.0), 

# Define the inference
embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embed')

# Construct variables for NCE loss
nce_weight = tf.Variable(tf.truncated_normal([VOCAB_SIZE, EMBED_SIZE],
                                            stddev=1.0 / (EMBED_SIZE ** 0.5)), 
nce_bias = tf.Variable(tf.zeros([VOCAB_SIZE]), name='nce_bias')

# Define loss function to be NCE loss function
loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight, 
                                    num_classes=VOCAB_SIZE), name='loss')

# Define optimizer
global_step = tf.Variable(0, name='global_step', trainable=False)
optimizer = tf.train.GradientDescentOptimizer(1.0).minimize(loss, global_step=global_step)
  • 2
    $\begingroup$ Just a guess -- Vanilla Adam updates all parameters at every step, while lazy Adam only updates parameters that are actually employed -- in a sparse setting like a language model, that can make a big difference, because lazy Adam applies no updates to rare words until they appear, at which time they get a big update. More common words are updated more frequently. $\endgroup$ – Sycorax Jul 4 '17 at 4:35
  • $\begingroup$ @Sycorax So you're saying that LazyAdam would do better since it doesn't needlessly update parameters it shouldn't be messing with? I'm having a little trouble understanding why the sparsity would affect it. $\endgroup$ – Puzzler3141 Jul 4 '17 at 4:39
  • 1
    $\begingroup$ To future reviewers: I'm not sure why this question was closed. It's asking about optimization, and my understanding is that optimization is on-topic here. $\endgroup$ – Sycorax Jul 4 '17 at 15:02
  • $\begingroup$ Possibly answered here: stats.stackexchange.com/questions/313278/… $\endgroup$ – Sycorax Sep 20 '18 at 14:56
  • $\begingroup$ It seems that Adam is not good for word embedding models in general, better to use Adagrad. hackernoon.com/… ruder.io/optimizing-gradient-descent/index.html#adagrad $\endgroup$ – SantoshGupta7 Aug 1 '19 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.