# Gradient descent for a noisy system

I have a system with tuning parameter $w$. To evaluate this system I use cost function $f(w)$.

I try finding the optimum value for $w$ using Gradient Descent starting from $w_0$.

The problem with this method is falling into the noise trap instead of the real minimum.

Without migrating to Genetic Algorithm, is there any solution under Gradient Descent to escape these local minimums due to noise?

It might be possible to use Noisy gradient descent and keep track of function value for instance the update would look like $$w^{k+1}\gets w^k-\eta \nabla f(w^k) + \epsilon$$ also you may use Adagrad for efficiently. It might be helpful to look at this post here, Also if your problem has stochastic gradient like setting, SGD or Adagrad with SGD should be work as they are inherently noisy.
• Here $\epsilon$ is drawn from a gaussian distribution, you can try setting $\mu$ = 0 and tune $\sigma$ the standard deviation to get best performance. But it should be done manually you can try doing a grid search for $\sigma$ in $(0,1]$ and run for each of the values and take the best one. I hope it answers your question – user52705 Mar 30 '16 at 9:22