1
$\begingroup$

I'm starting out learning about gradient descent, and have a couple of conceptual questions.

  1. I noticed a common pitfall of gradient descent is getting stuck in the local optima. How can this be avoided (while still being considered gradient descent)?

  2. If a function alternates up and down infinitely across the x axis, does that mean it has an infinite amount of local optima?

Example:

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Comments. Your point (1) is why convex problems tend to be solvable why non-convex problems can be a complete disaster. The answer to (2) is yes. $\endgroup$ – Matthew Gunn Oct 6 '16 at 0:06
1
$\begingroup$
  1. run it multiple times from different starting points. Momentum. Pretraining. those are the methods I can think of.

  2. It's the shape of the error function that is important and that probably won't be a sinusoid. If you are fitting a sinusoid with squared error, then your function would be nice and convex with no local mina

$\endgroup$
2
  • $\begingroup$ I don't understand your answer to the second part. I know a function like that is unlikely, but I would still like to know if what that means for the local optima. I understand maxima and minima are peaks and valleys, but what if there are an infinite amount of peaks and valleys? Does that mean there are infinite optima? $\endgroup$ – Bob Oct 6 '16 at 0:00
  • $\begingroup$ Imagine that this function has minimum at 0. That picture would mean that you are fitting a function with one parameter (x axis) and the error corresponding to this size of a parameter is on a y axis. In that case yes, you will have infinitely many local minima. $\endgroup$ – rep_ho Oct 6 '16 at 0:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.