0
$\begingroup$

Recall the Adam update rule:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$\hat{m}_t = \dfrac{m_t}{1 - \beta^t_1}$$ $$\hat{v}_t = \dfrac{v_t}{1 - \beta^t_2} $$ $$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

Suppose for all steps $t>t^*$, the gradient $g_t$ is $0$. Then we have $$\theta_{t^*+k+1} = \theta_{t^*+k} - \eta \cdot \frac{\frac{\beta_1^k m}{1-\beta_1^{k+x}}}{\sqrt\frac{\beta_2^k v}{1-\beta_2^{k+x}}+\epsilon}$$

However, I don't see any guarantee that $\frac{\frac{\beta_1^k m}{1-\beta_1^{k+x}}}{\sqrt\frac{\beta_2^k v}{1-\beta_2^{k+x}}+\epsilon}$ must go to zero for large $k$. Is there any literature suggesting this? I would expect that in such a regime, Adam should stop making updates of significant magnitude.

$\endgroup$
1
$\begingroup$

In Adam there's epsilon parameter, which tries to prevent that, so you'll always be doing some steps. As we're operating in non-convex spaces, halting optimizer in local optimum wouldn't neceserally be a good idea.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.