2
$\begingroup$

I have a system with tuning parameter $w$. To evaluate this system I use cost function $f(w)$.

I try finding the optimum value for $w$ using Gradient Descent starting from $w_0$.

The problem with this method is falling into the noise trap instead of the real minimum.

Without migrating to Genetic Algorithm, is there any solution under Gradient Descent to escape these local minimums due to noise?

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It might be possible to use Noisy gradient descent and keep track of function value for instance the update would look like $$w^{k+1}\gets w^k-\eta \nabla f(w^k) + \epsilon$$ also you may use Adagrad for efficiently. It might be helpful to look at this post here, Also if your problem has stochastic gradient like setting, SGD or Adagrad with SGD should be work as they are inherently noisy.

Improve this answer
$\endgroup$
2
  • $\begingroup$ When I add noise, shouldnt I check if noise is improving the system otherwise I should remove that particular noise? $\endgroup$
    – ar2015
    Commented Mar 30, 2016 at 8:41
  • $\begingroup$ Here $\epsilon$ is drawn from a gaussian distribution, you can try setting $\mu$ = 0 and tune $\sigma$ the standard deviation to get best performance. But it should be done manually you can try doing a grid search for $\sigma$ in $(0,1]$ and run for each of the values and take the best one. I hope it answers your question $\endgroup$
    – user52705
    Commented Mar 30, 2016 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.