I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following:

I am performing optimization over hundreds of parameters, each fed into a logistic function to yield a final value between 0 and 1. I have tried performing gradient descent using various techniques like iRPROP+, but keep getting stuck in local minima despite restarting the search at multiple points in the search space.

This makes me believe I need to use a global optimization technique that can escape from local minima. However, I would like to make use of the gradient of the function, which is available and can be used to speed up search.

Overfitting is not an issue in the problem I am tackling.

  • 1
    $\begingroup$ Sounds like this may be a use-case for Stochastic Gradient Hamiltonian Monte Carlo. arxiv.org/abs/1402.4102 $\endgroup$ – Sycorax Aug 30 '17 at 15:52
  • $\begingroup$ @Sycorax: I found the title of the paper you cited funny (from an Optimisation point of view) and just for laughs I searched "Stochastic Quasi-Newton HMC." and apparently it exists! "Stochastic Quasi-Newton Langevin Monte Carlo". I am taking bets for when the "Box-constraints Stochastic Optimisation by Quadratic Approximations" will appear. :D (In all seriousness, these seem to be addressing important questions, I am just a bit surprised they are publication venues are not "classical" Optimisation publications.) $\endgroup$ – usεr11852 Sep 3 '17 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.