Say I have some data and some graph fitting function. Just using gradient descent will find one particular local solution.
Now what I want to do is also put "feelers" out for other solutions. In essence I want to run the gradient descent many times in parallel to find different local solutions. And select the best one.
Once it has one, it could put it's feelers out again, by running another gradient descent with a random difference in variables. (Say a short difference, a medium distance, and a long distance).
Then it could run all these other trials and see if it can find another local minimum. (But keeping the local minimum it already found).
This way, hopefully over time it can eventually arrive at the best fit.
To visualise this imagine climbing to the top of a mountain. Then dropping people randomly around that mountain to see if they climb up to another mountain that is higher.
Well my question is, what is this kind of procedure called? In APIs like tensor flow would you have to implement something like this yourself?
Another related question is that in higher dimensions, are things less likely to get stuck in local minima? So this is not needed?