I am implementing an autoencoder. The cost function I am trying to minimize is mean square error. When I use RMSPropOptimizer, the output is good. But with GradientDescentOptimizer, the output is not good. So can anyone please explain the differences in between these two optimizer ?


GradientDescentOptimizer implements the most basic version of Gradient Descent while RMSPropOptimizer implements an adaptive version of Stochastic Gradient Descent called RMS Prop "Root Mean Squared Propagation.

Basic gradient descent is the slowest of any of the neural network optimizers, you will want to run it for a much longer number of epochs if you want to achieve results similar to RMSProp. In fact just about anyone of the other methods:

tf.train.AdagradOptimizer, tf.train.AdagradDAOptimizer, tf.train.MomentumOptimizer, tf.train.AdamOptimizer, tf.train.FtrlOptimizer,...

will give you better results than gradient descent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.