9
$\begingroup$

I've learned from DL classes that Adam should be the default choice for neural network training. However, I've recently seen more and more recent reinforcement learning agents use RMSProp instead of Adam as their optimizer, such as FTW from DeepMind. I'm wondering when prefer RMSProp to Adam and when the other way around?

I know the background math of both algorithms. To my best knowledge, Adam improves RMSProp by including momentum and bias correction. As a result, I personally preferred Adam as default choice. But I began to waver between RMSProp and Adam these day. Hope someone could give a more comprehensive guide on these optimization algorithms(better with some intuition about how to tune the hyperparameters, such as momentum). Thanks in advance!

Improve this question
$\endgroup$
4
  • $\begingroup$ Deepmind has always had a thing for RMSprop... I've heard from an optimization expert that RMSprop is most suited for "sparse" problems, but I'm not how that applies to NNs (which have pretty dense connectivity). I think mainly, if you use one optimizer for long enough, you start to gain a sixth sense for exactly how to tune the hyperparameters to get it to work really well. And then after that, you end up reluctant to switch -- explaining why some authors always use RMSprop and some always use Adam. $\endgroup$
    – shimao
    Commented Nov 30, 2019 at 22:31
  • $\begingroup$ Thanks for answering. If RMSprop is suitable for sparse problems, then in RL if I understand the "sparse problems" right, it may be a better choice for sparse reward problems. But do you have any idea why it is more suitable for sparse problems? $\endgroup$
    – Maybe
    Commented Dec 1, 2019 at 23:59
  • $\begingroup$ unfortunately no -- was just a passing comment. $\endgroup$
    – shimao
    Commented Dec 2, 2019 at 4:14
  • $\begingroup$ Thanks, anyway. At least now I know RMSprop may be more suitable for sparse reward problems. I'll keep that in mind. $\endgroup$
    – Maybe
    Commented Dec 2, 2019 at 8:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.