I am implementing an autoencoder. The cost function I am trying to minimize is mean square error. When I use RMSPropOptimizer, the output is good. But with GradientDescentOptimizer, the output is not good. So can anyone please explain the differences in between these two optimizer ?
GradientDescentOptimizer implements the most basic version of Gradient Descent while RMSPropOptimizer implements an adaptive version of Stochastic Gradient Descent called RMS Prop "Root Mean Squared Propagation.
Basic gradient descent is the slowest of any of the neural network optimizers, you will want to run it for a much longer number of epochs if you want to achieve results similar to RMSProp. In fact just about anyone of the other methods:
tf.train.AdagradOptimizer, tf.train.AdagradDAOptimizer, tf.train.MomentumOptimizer, tf.train.AdamOptimizer, tf.train.FtrlOptimizer,...
will give you better results than gradient descent.