# Questions about gradient descent and local optima

I'm starting out learning about gradient descent, and have a couple of conceptual questions.

1. I noticed a common pitfall of gradient descent is getting stuck in the local optima. How can this be avoided (while still being considered gradient descent)?

2. If a function alternates up and down infinitely across the x axis, does that mean it has an infinite amount of local optima?

Example: • Comments. Your point (1) is why convex problems tend to be solvable why non-convex problems can be a complete disaster. The answer to (2) is yes. – Matthew Gunn Oct 6 '16 at 0:06

## 1 Answer

1. run it multiple times from different starting points. Momentum. Pretraining. those are the methods I can think of.

2. It's the shape of the error function that is important and that probably won't be a sinusoid. If you are fitting a sinusoid with squared error, then your function would be nice and convex with no local mina

• I don't understand your answer to the second part. I know a function like that is unlikely, but I would still like to know if what that means for the local optima. I understand maxima and minima are peaks and valleys, but what if there are an infinite amount of peaks and valleys? Does that mean there are infinite optima? – Bob Oct 6 '16 at 0:00
• Imagine that this function has minimum at 0. That picture would mean that you are fitting a function with one parameter (x axis) and the error corresponding to this size of a parameter is on a y axis. In that case yes, you will have infinitely many local minima. – rep_ho Oct 6 '16 at 0:11