I am trying to understand gradient descent algorithm by plotting the error vs value of parameters in the function. What would be an example of a simple function of the form y = f(x) with just just one input variable x and two parameters w1 and w2 such that its loss function has multiple minima? What i am trying to achieve is something similar to this :
How does one know if the function has multiple minima without plotting the graph ? What branch of mathematics deals with this topic ?
Regarding example of functions with multiple local minima I would suggest visiting a website like the Virtual Library of Simulation Experiments: Test Functions and Datasets - Optimization Test Problems from Simon Fraser University. It contains many examples of functions with many local minima. A trivial two-factor example would be something like: $x \sin(w_1 x+w_2)$. In real-life terms most functions that might reflect some seasonality/periodicity will potentially have multiple local minima relating to that seasonal/periodic effect.
The most straightforward way to asses if a particular function has multiple local minima is to use calculus. Multiple local minima would relate to multiple instances of first derivatives being zero and second derivatives being positive. As Neil mentioned: "in two dimensions (like the plot he's drawn), the second derivative is a matrix, in which case a minimum corresponds to a positive definite second derivative matrix." Moving to multivariate functions will be reflected in dimensions of the function's derivatives. The object we use in that case is the Hessian matrix (which has mentioned we want to be at least positive semi-definite).
The branch of mathematics dealing with topic is called Mathematical Optimisation. Real-life examples of optimisation tasks are extensively involved in the field of Operational Research.
The functions that you are looking for are known as test functions or artificial landscapes.
Wikipedia has a very nice list of test functions for optimization. I recommend looking at the references from the link directly.
If they are only a few and you can estimate a range where they will lay, you can try descent methods with different starting points that will converge to each of them.
This practice works some of the time, but as we increase the number of dimensions (or as we know less about the shape of the function), this simplistic approach will no longer be practical (see previous answers)
