I can implement a vanilla gradient descent alogrithm, to minimize some function $f(x,y)$.

Now if I know a priori that this function is smooth with a single global minimum (i.e. no local minima for a gradient descent algorithm to get stuck in) what is the best gradient descent variation to use in terms of speed?

I have found a description of different approaches here, but am unclear which way would be optimal.


  • 1
    $\begingroup$ the answer clearly depends on the function and what you mean by "smooth." For instance, is a function with continuous first derivative and discontinuous second smooth? $\endgroup$ – Aksakal Mar 14 '18 at 14:25
  • $\begingroup$ Sure. What I mean is some simple quadratic/bowl function. $\endgroup$ – user1887919 Mar 14 '18 at 14:35
  • 1
    $\begingroup$ If it's quadratic, why not quadratic optimization? foes it have to be gradient descent? Also, look at Newton method, it works best for quadratic functions $\endgroup$ – Aksakal Mar 14 '18 at 14:43
  • $\begingroup$ It you are minimizing a convex quadratic with no constraints, then the global minimum can be found with a linear equation solve. $\endgroup$ – Mark L. Stone Mar 14 '18 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.