I'm currently trying to understand SGD and ADAM optimization, and I understand that ADAM optimization performs well on non-convex loss functions and that SGD performs well on convex loss functions (correct me if I'm wrong), but I'm unsure as to why? What is going on in the math that makes ADAM do well in non-convex functions?

And does ADAM do bad on convex functions? Or is ADAM pretty much always preferred over SGD, for instance?

If you could include an intuitive explanation and a more mathematical explanation, I would really appreciate it since I am trying to understand the math behind ADAM but am having trouble.


1 Answer 1


SGD provably converges for convex functions. However, people discovered that it's also pretty decent for training neural networks which have very non-convex high dimensional loss functions.

Soon, people discovered different tweaks and additions to SGD which make it even better at training neural networks. One of these tweaks is called ADAM. These tweaked versions of SGD still converge on convex functions, except..

Later on, some people found mistakes in the original convergence analysis of Adam and similar optimizers. Reddi et al even constructed some explicit convex functions on which Adam doesn't converge, and proposed a fix, called "amsgrad". However, despite theoretical benefits, amsgrad doesn't seem to have caught on - hearsay suggests it doesn't actually improve performance when training NNs.

On convex functions, it probably doesn't matter very much which optimizer you use. You could construct some pathological function for which Adam does not converge, but I'm not sure if it's a practical issue. Of course, things like momentum (which Adam/amsgrad has) is still beneficial for convex functions.

For training neural networks, Adam (or related optimizers) are still the gold standard. SGD with momentum is also commonly employed - some people say it generalizes better, and it also consumes slightly less memory as there's no need to track the second moment. Plain SGD is probably not very good.

TLDR: Adam is good because people tweaked SGD to make it perform better, but this is not specific to convex or nonconvex functions. But those tweaks (accidentally) made it perform badly on some pathological cases (which can appear in both convex and nonconvex functions). But practically speaking, this doesn't seem to matter much.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.