SGD provably converges for convex functions. However, people discovered that it's also pretty decent for training neural networks which have very non-convex high dimensional loss functions.
Soon, people discovered different tweaks and additions to SGD which make it even better at training neural networks. One of these tweaks is called ADAM. These tweaked versions of SGD still converge on convex functions, except..
Later on, some people found mistakes in the original convergence analysis of Adam and similar optimizers. Reddi et al even constructed some explicit convex functions on which Adam doesn't converge, and proposed a fix, called "amsgrad". However, despite theoretical benefits, amsgrad doesn't seem to have caught on - hearsay suggests it doesn't actually improve performance when training NNs.
On convex functions, it probably doesn't matter very much which optimizer you use. You could construct some pathological function for which Adam does not converge, but I'm not sure if it's a practical issue. Of course, things like momentum (which Adam/amsgrad has) is still beneficial for convex functions.
For training neural networks, Adam (or related optimizers) are still the gold standard. SGD with momentum is also commonly employed - some people say it generalizes better, and it also consumes slightly less memory as there's no need to track the second moment. Plain SGD is probably not very good.
TLDR: Adam is good because people tweaked SGD to make it perform better, but this is not specific to convex or nonconvex functions. But those tweaks (accidentally) made it perform badly on some pathological cases (which can appear in both convex and nonconvex functions). But practically speaking, this doesn't seem to matter much.
https://openreview.net/forum?id=ryQu7f-RZ
