Deep learning literature is full of clever tricks with using non-constant learning rates in gradient descent. Things like exponential decay, RMSprop, Adagrad etc. are easy to implement and are available in every deep learning package, yet they seem to be nonexistent outside of neural networks. Is there any reason for this? If it is that people simply don't care, is there a reason why don't we have to care outside of neural networks?

  • 2
    $\begingroup$ I think line search or trust region method are "non-constant" learning rates. $\endgroup$ – Haitao Du Mar 2 '18 at 17:58
  • 2
    $\begingroup$ There are lots of non-constant gradient methods that were developed independently of NNs. Barzilai-Borwein GD and Nesterov GD are two prominent examples. $\endgroup$ – Sycorax Mar 2 '18 at 18:19
  • $\begingroup$ @Sycorax but are they actually used on daily basis outside of NNs? $\endgroup$ – Tim Mar 2 '18 at 18:27
  • 2
    $\begingroup$ @Tim I can't say. When I need to do local search outside of NNs, I have the luxury of using second-order methods. But I was excited to learn about faster GD methods for the occasion that I might have a cute trick in my back pocket. $\endgroup$ – Sycorax Mar 2 '18 at 18:29
  • 1
    $\begingroup$ It is worth noting that (to my surprise) I have come across cases that GBMs do not use constant learning rates, somewhat to the surprise of people. A particular example has been the implementation of DART on LightGBM. While the original papers do not use a increasingly smaller LR the actual implementation does by default. $\endgroup$ – usεr11852 Sep 9 '19 at 16:19

Disclaimer: I don't have so much experience with optimization outside of neural networks, so my answer will be clearly biased, but there are several things that play role:

  • (Deep) neural networks have a lot of parameters. This has several implications:

    Firstly, it kind-of rules out higher order methods simply because computing Hessian and higher derivatives becomes infeasible. In other domains, this may be a valid approach better than any tweaks to SGD.

    Secondly, although SGD is wonderful, it tends to be impractically slow. These improved SGD variants mainly enable faster training, while potentially losing some of the nice properties of SGD. In other domains, the SGD training time may not be the bottleneck, so improvements gained by speeding it up may be simply negligible.

  • Training (deep) neural networks is non-convex optimization and I am not aware of significant convex relaxation results in the field. Unlike other fields, neural networks are not focused on provably globally optimal solutions, which leads to investing more efforts into improving the properties of loss surface and its traversal during the optimization.

    In other fields, employing convex relaxation and obtaining globally optimal solutions may be in the center of the interest instead of the optimization algorithm, because once the problem is defined as a convex problem, the choice of the optimization algorithm cannot improve the quality of the solution.

I suppose this answer does not cover all possible aspects and I am myself curious about other opinions.

| cite | improve this answer | |
  • $\begingroup$ So you are basically saying that other problems are much simpler, so don't need the tricks and vanilla SGD is enough for them? $\endgroup$ – Tim Mar 2 '18 at 18:31
  • 3
    $\begingroup$ That is oversimplifying my message. 1) some problems can use higher order methods, no need for adaptive SGD. 2) some problems cannot benefit from SGD improvement due to Amdahl's law. 3) some problems may offer convex solutions and the main difficulty is in posing them as convex. None of these says other problems are much simpler than deep learning, rather explains why improving SGD is not in the center of their attention. $\endgroup$ – Jan Kukacka Mar 2 '18 at 18:39
  • $\begingroup$ A possible point 4: if you took some other method and made it complex enough (high dimensional, nonlinear, nonconvex) to benefit from sophisticated gradient descent methods, it would probably be called a neural network. $\endgroup$ – Nathaniel Mar 3 '18 at 3:04
  • 1
    $\begingroup$ @JanKukacka I know, I was looking for clarification since your answer was indirect $\endgroup$ – Tim Mar 3 '18 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.