I have a system with tuning parameter $w$. To evaluate this system I use cost function $f(w)$.

I try finding the optimum value for $w$ using Gradient Descent starting from $w_0$.

The problem with this method is falling into the noise trap instead of the real minimum.

Without migrating to Genetic Algorithm, is there any solution under Gradient Descent to escape these local minimums due to noise?

enter image description here


It might be possible to use Noisy gradient descent and keep track of function value for instance the update would look like $$w^{k+1}\gets w^k-\eta \nabla f(w^k) + \epsilon$$ also you may use Adagrad for efficiently. It might be helpful to look at this post here, Also if your problem has stochastic gradient like setting, SGD or Adagrad with SGD should be work as they are inherently noisy.

  • $\begingroup$ When I add noise, shouldnt I check if noise is improving the system otherwise I should remove that particular noise? $\endgroup$ – ar2015 Mar 30 '16 at 8:41
  • $\begingroup$ Here $\epsilon$ is drawn from a gaussian distribution, you can try setting $\mu$ = 0 and tune $\sigma$ the standard deviation to get best performance. But it should be done manually you can try doing a grid search for $\sigma$ in $(0,1]$ and run for each of the values and take the best one. I hope it answers your question $\endgroup$ – user52705 Mar 30 '16 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.