1
$\begingroup$

I am using ADAM as an optimization algorithm to minimize some black box function $f(x,y)$. I know this function is convex and has a minimum $f(5,5) = 0$.

Initially, the algorithm proceeds as expected:

enter image description here

where the $x$ and $y$ axis are the respective variables, and the red dot indicates the target of $x=5, y=5$

However at small values of $f(x,y)$ I get,

enter image description here

This is a problem if I want convergence to zero within some precision $\epsilon \ \sim 10^{-12}$.

What is the cause of oscillations like this?

What is a solution to avoid/compensate for them?

$\endgroup$
1
$\begingroup$

What is the cause of oscillations like this?

SGD (with and without momentum) can naturally have oscillations. This is largely due to high curvature areas of the objective landscape. In particular, especially with momentum present, gradient descent behaves like a damped harmonic oscillator.

Some references:

What is a solution to avoid/compensate for them?

One answer is to correct for this curvature, which is presumably high near the minimum. The first reference above suggests tuning the gradient descent hyper-parameters. Another option, depending on your situation (mostly the size of the model and the inherent noise in the gradient estimates), it may be possible to use second-order optimization methods instead, such as the natural gradient (see also KFAC) or Newton's method (e.g. see this, but also see here).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.