I am performing experiments on the EMNIST validation set using networks with RMSProp, Adam and SGD. I am achieving 87% accuracy with SGD(learning rate of 0.1) and dropout (0.1 dropout prob) as well as L2 regularisation (1e-05 penalty). When testing the same exact configuration with RMSProp and Adam as well as the initial learning rate of 0.001, I am achieving accuracy of 85% and a significantly less smooth training curve. I do not know how to explain this behaviour. What can be the reason behind the lack of smoothness in the training curve and the lower accuracy and higher error rates achieved?
$\begingroup$
$\endgroup$
5
-
$\begingroup$ This depends on the network. Can you show us details about the network? Also can you provide the learning curves? $\endgroup$– MemmingCommented Nov 26, 2017 at 16:27
-
$\begingroup$ This is a network with 5 layers (Dropout, Affine, ELU in each layer), set up as follows: 150 hidden dimensions, ELU activation function used, 0.1 learning rate for SGD, 0.001 learning rate for RMS and Adam, L2 regularisation with 1e-05 penalty, Dropout with 0.1 exclusion probability. $\endgroup$– AlkCommented Nov 26, 2017 at 16:29
-
$\begingroup$ And when you say "exact same configuration ... initial learning rate of 0.001" do you mean you used a different learning rate or you did two experiments: one with the same learning rate and one with a different one? It may depend on the actual software you're using as to what parameters default to what. $\endgroup$– WayneCommented Nov 26, 2017 at 16:30
-
$\begingroup$ So I used 0.1 for SGD and 0.001 for both Adam and RMSProp. This is because when I ran Adam and RMSProp with 0.1 learning rate they both performed badly with an accuracy of 60%. Also, 0.001 is the recommended value in the paper on Adam. $\endgroup$– AlkCommented Nov 26, 2017 at 16:32
-
$\begingroup$ You should post learning curves, and whether you measure error on test or training data. $\endgroup$– Jakub BartczukCommented May 30, 2018 at 8:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
After researching a few articles online and Keras documentation it is suggested that the RMSProp optimizer is recommended for recurrent neural networks.https://github.com/keras-team/keras/blob/master/keras/optimizers.py#L209
Stochastic Gradient Descent seems to take advantage of its learning rate and momentum between each batch to optimize the model’s weights based on the information of the loss function in my case is 'categorical_crossentropy'.
I suggest https://www.ruder.io/optimizing-gradient-descent/ for additional information about optimization algorithms.
-
$\begingroup$ I am not sure what do you mean by "RMSProp optimizer is recommended for recurrent neural networks". The link you put is now broken and it is nowhere mentioned that it is recommended. $\endgroup$ Commented Dec 3, 2019 at 19:16
-
$\begingroup$ For the life of me I can't remember which paper it was in, but I have also read the advice that RMSProp is recommended for RNNs. $\endgroup$ Commented Jul 16, 2020 at 0:10
-
$\begingroup$ They removed this part of the Keras documentation: github.com/keras-team/keras/issues/12460 $\endgroup$– hadrienjCommented Nov 5, 2020 at 11:19