Gradient descent with random kicks on non-convex functions I've recently learned gradient descent and it clearly gets stuck at local minimums when applied to non-convex functions.
Can't we just randomly kick the values in between steps when iterating?
Kind of like quantum tunneling. That would drastically increase the probability to reach global minimum.
 A: {2} mentions to well-known methods:

ITERATIVE HILL-CLIMBING: This method is devised by combining
  hill-climbing and random search. Once one peak has been found, the
  hillclimbing process is repeated again at another randomly selected
  location. This search is performed in isolation hence, it takes no
  account of the overall idea of the shape of the problem domain. 
SIMULATED ANNEALING: Invented in 1982 by Kirkpatrick , it is
  essentially a modified version of hill-climbing. Starting from a
  random point in the search space, a random move is made. If the move
  takes us to a higher point, it is accepted, otherwise it is accepted
  with the probability decresaing with time. The probabilty starts from
  1, decreasing towards zero as time passes. As such, any move is
  accepted at the begining, but as the probability continues to
  decrease, the probability of accepting negative moves are lowered.
  Negative moves are essential sometimes, if local maxima are to be
  escaped, but too many negative moves would lead the search away from
  the maxima. Again this method deals with one candidate one at a time
  and so does not build an overall picture of the search space, and no
  information from previous moves is used to guide the selection of new
  moves. This technique has been successful in many applications, for
  example VLSI circuit layout.

You can also use an evolutionary algorithm and add some hill climbing (= gradient ascent if your function is differentiable), as done in {1}:


References:


*

*{1} Su, Shih-Chieh, Cheng-Jian Lin, and Chuan-Kang Ting. "An effective hybrid of hill climbing and genetic algorithm for 2D triangular protein structure prediction." Proteome science 9, no. 1 (2011): 1. https://scholar.google.com/scholar?cluster=8852775032391122915&hl=en&as_sdt=0,22 ; https://doi.org/10.1186/1477-5956-9-S1-S19

*{2} https://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol2/hmw/article2.html
