I was implementing word2vec in TensorFlow and found that Gradient Descent worked much better and faster than the AdamOptimizer. I was under the impression that Adam was the "smarter" option that almost always does better than GD. I used several starting learning rates for Adam, from 1.0 to 0.01, but none did nearly as well as GD with a learning rate of 1.0. Am I missing something about these optimizers or their application to word2vec in particular?
# Define the placeholders for input and output center_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE], name='center_words') target_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE, 1], name='target_words') # Define weights. In word2vec, it's actually the weights that we care about embed_matrix = tf.Variable(tf.random_uniform([VOCAB_SIZE, EMBED_SIZE], -1.0, 1.0), name='embed_matrix') # Define the inference embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embed') # Construct variables for NCE loss nce_weight = tf.Variable(tf.truncated_normal([VOCAB_SIZE, EMBED_SIZE], stddev=1.0 / (EMBED_SIZE ** 0.5)), name='nce_weight') nce_bias = tf.Variable(tf.zeros([VOCAB_SIZE]), name='nce_bias') # Define loss function to be NCE loss function loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight, biases=nce_bias, labels=target_words, inputs=embed, num_sampled=NUM_SAMPLED, num_classes=VOCAB_SIZE), name='loss') # Define optimizer global_step = tf.Variable(0, name='global_step', trainable=False) optimizer = tf.train.GradientDescentOptimizer(1.0).minimize(loss, global_step=global_step)