5
$\begingroup$

I have been training a Spatial Transformer network with DNN on GTRSB dataset. I initially used SGF with momentum and was able to achieve good accuracy.

For further improvements and testing, I decided to change the optimizer to Adam, but strangely I am not seeing any increasing in training or validation accuracy after quite a many epochs.

Is it possible that Adam is not well suited for this dataset? (or in general is it possible for optimizers to work on one dataset but work fine on others?)

Edit: I tried running Adam on a smaller dataset with lower learning rate (initial was 0.01, now I set it to 0.001). On smaller one, it should some increase in accuracy, but on the bigger dataset the issue still persists.

Edit 2: Further decreasing the learning rate to 0.0001 makes even Adam work on larger dataset.

Thanks

$\endgroup$
3
  • $\begingroup$ My understanding is that different optimizers (adam vs sgd) won't necessarily give you a better answer (i.e. better accuracy), but rather will just converge faster (with smaller number of steps). $\endgroup$
    – sega_sai
    Nov 12, 2017 at 3:09
  • $\begingroup$ @sega_sai Yes, I understand that, but my issue is that Adam is not even optimizing on larger dataset, I have made an edit based on testing I have done. $\endgroup$
    – apsdehal
    Nov 12, 2017 at 3:25
  • $\begingroup$ What is GTRSB? What is SGF? $\endgroup$
    – Sycorax
    Jul 16, 2018 at 0:12

1 Answer 1

8
$\begingroup$

The benefits of Adam can be marginal, at best. The initial results were strong, but there is evidence that Adam converges to dramatically different minima compared to SGD (or SGD + momentum).

"The Marginal Value of Adaptive Gradient Methods in Machine Learning" Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple over-parameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

Speaking from personal experience, Adam can struggle unless you set a small learning rate -- which sort of defeats the whole purpose of using an adaptive method in the first place, not to mention all of the wasted time spent toying with learning rate.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.