I was implementing word2vec in TensorFlow and found that Gradient Descent worked much better and faster than the AdamOptimizer. I was under the impression that Adam was the "smarter" option that almost always does better than GD. I used several starting learning rates for Adam, from 1.0 to 0.01, but none did nearly as well as GD with a learning rate of 1.0. Am I missing something about these optimizers or their application to word2vec in particular?
Code:
# Define the placeholders for input and output
center_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE], name='center_words')
target_words = tf.placeholder(tf.int32, shape=[BATCH_SIZE, 1], name='target_words')
# Define weights. In word2vec, it's actually the weights that we care about
embed_matrix = tf.Variable(tf.random_uniform([VOCAB_SIZE, EMBED_SIZE], -1.0, 1.0),
name='embed_matrix')
# Define the inference
embed = tf.nn.embedding_lookup(embed_matrix, center_words, name='embed')
# Construct variables for NCE loss
nce_weight = tf.Variable(tf.truncated_normal([VOCAB_SIZE, EMBED_SIZE],
stddev=1.0 / (EMBED_SIZE ** 0.5)),
name='nce_weight')
nce_bias = tf.Variable(tf.zeros([VOCAB_SIZE]), name='nce_bias')
# Define loss function to be NCE loss function
loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight,
biases=nce_bias,
labels=target_words,
inputs=embed,
num_sampled=NUM_SAMPLED,
num_classes=VOCAB_SIZE), name='loss')
# Define optimizer
global_step = tf.Variable(0, name='global_step', trainable=False)
optimizer = tf.train.GradientDescentOptimizer(1.0).minimize(loss, global_step=global_step)
