I was implementing word2vec in TensorFlow and found that Gradient Descent worked much better and faster than the AdamOptimizer. I was under the impression that Adam was the "smarter" option that almost always does better than GD. I used several starting learning rates for Adam, from 1.0 to 0.01, but none did nearly as well as GD with a learning rate of 1.0. Am I missing something about these optimizers or their application to word2vec in particular?


# Define the placeholders for input and output
center_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE], name='center_words')
target_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE, 1], name='target_words')

# Define weights. In word2vec, it's actually the weights that we care about
embed_matrix = tf.Variable(tf.random_uniform([VOCAB_SIZE, EMBED_SIZE], -1.0, 1.0), 

# Define the inference
embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embed')

# Construct variables for NCE loss
nce_weight = tf.Variable(tf.truncated_normal([VOCAB_SIZE, EMBED_SIZE],
                                            stddev=1.0 / (EMBED_SIZE ** 0.5)), 
nce_bias = tf.Variable(tf.zeros([VOCAB_SIZE]), name='nce_bias')

# Define loss function to be NCE loss function
loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight, 
                                    num_classes=VOCAB_SIZE), name='loss')

# Define optimizer
global_step = tf.Variable(0, name='global_step', trainable=False)
optimizer = tf.train.GradientDescentOptimizer(1.0).minimize(loss, global_step=global_step)
  • 2
    $\begingroup$ Just a guess -- Vanilla Adam updates all parameters at every step, while lazy Adam only updates parameters that are actually employed -- in a sparse setting like a language model, that can make a big difference, because lazy Adam applies no updates to rare words until they appear, at which time they get a big update. More common words are updated more frequently. $\endgroup$
    – Sycorax
    Jul 4, 2017 at 4:35
  • $\begingroup$ @Sycorax So you're saying that LazyAdam would do better since it doesn't needlessly update parameters it shouldn't be messing with? I'm having a little trouble understanding why the sparsity would affect it. $\endgroup$ Jul 4, 2017 at 4:39
  • 1
    $\begingroup$ To future reviewers: I'm not sure why this question was closed. It's asking about optimization, and my understanding is that optimization is on-topic here. $\endgroup$
    – Sycorax
    Jul 4, 2017 at 15:02
  • $\begingroup$ Possibly answered here: stats.stackexchange.com/questions/313278/… $\endgroup$
    – Sycorax
    Sep 20, 2018 at 14:56
  • $\begingroup$ It seems that Adam is not good for word embedding models in general, better to use Adagrad. hackernoon.com/… ruder.io/optimizing-gradient-descent/index.html#adagrad $\endgroup$ Aug 1, 2019 at 4:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.