I am performing experiments on the EMNIST validation set using networks with RMSProp, Adam and SGD. I am achieving 87% accuracy with SGD(learning rate of 0.1) and dropout (0.1 dropout prob) as well as L2 regularisation (1e-05 penalty). When testing the same exact configuration with RMSProp and Adam as well as the initial learning rate of 0.001, I am achieving accuracy of 85% and a significantly less smooth training curve. I do not know how to explain this behaviour. What can be the reason behind the lack of smoothness in the training curve and the lower accuracy and higher error rates achieved?

  • $\begingroup$ This depends on the network. Can you show us details about the network? Also can you provide the learning curves? $\endgroup$ – Memming Nov 26 '17 at 16:27
  • $\begingroup$ This is a network with 5 layers (Dropout, Affine, ELU in each layer), set up as follows: 150 hidden dimensions, ELU activation function used, 0.1 learning rate for SGD, 0.001 learning rate for RMS and Adam, L2 regularisation with 1e-05 penalty, Dropout with 0.1 exclusion probability. $\endgroup$ – Alk Nov 26 '17 at 16:29
  • $\begingroup$ And when you say "exact same configuration ... initial learning rate of 0.001" do you mean you used a different learning rate or you did two experiments: one with the same learning rate and one with a different one? It may depend on the actual software you're using as to what parameters default to what. $\endgroup$ – Wayne Nov 26 '17 at 16:30
  • $\begingroup$ So I used 0.1 for SGD and 0.001 for both Adam and RMSProp. This is because when I ran Adam and RMSProp with 0.1 learning rate they both performed badly with an accuracy of 60%. Also, 0.001 is the recommended value in the paper on Adam. $\endgroup$ – Alk Nov 26 '17 at 16:32
  • $\begingroup$ You should post learning curves, and whether you measure error on test or training data. $\endgroup$ – Jakub Bartczuk May 30 '18 at 8:36

After researching a few articles online and Keras documentation it is suggested that the RMSProp optimizer is recommended for recurrent neural networks.https://github.com/keras-team/keras/blob/master/keras/optimizers.py#L209

Stochastic Gradient Descent seems to take advantage of its learning rate and momentum between each batch to optimize the model’s weights based on the information of the loss function in my case is 'categorical_crossentropy'.

I suggest http://ruder.io/optimizing-gradient-descent/index.html for additional information about optimization algorithms.

  • $\begingroup$ I am not sure what do you mean by "RMSProp optimizer is recommended for recurrent neural networks". The link you put is now broken and it is nowhere mentioned that it is recommended. $\endgroup$ – Harshal Parekh Dec 3 '19 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.