# The correct implementation of momentum method and NAG

Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of initialization and momentum in deep learning, (Sutskever et al., 2013) which has a really nice explanation on it. But I realized that the algorithm was different compared to that on most websites such as http://ruder.io/optimizing-gradient-descent/index.html#momentum

\begin{aligned} v_{t+1} &=\mu v_{t}-\eta \nabla f\left(\theta_{t}\right) \\ \theta_{t+1} &=\theta_{t}+v_{t+1} \end{aligned}

And here's the one used widely, \begin{aligned} v_{t+1} &=\mu v_{t}+\eta \nabla_{\theta} J(\theta) \\ \theta_{t+1} &=\theta_{t}-v_{t+1} \end{aligned}

Clearly, they are different as the $$\mu v_{t}$$ term is added in the first update while being substracted in the second. Similarily for NAG updates, the paper implements

\begin{aligned} v_{t+1} &=\mu v_{t}-\eta \nabla f\left(\theta_{t}+\mu v_{t}\right) \\ \theta_{t+1} &=\theta_{t}+v_{t+1} \end{aligned}

While the commonly used version is,

\begin{aligned} v_{t+1} &=\mu v_{t}+\eta \nabla_{\theta} J\left(\theta-\eta v_{t}\right) \\ \theta_{t+1} &=\theta_{t}-v_{t+1} \end{aligned}

I have tried the implemented the second algorithm in the Coursera course and it works nicely but surely the one in the paper has also been implemented. So, are both correct and or the newer version has some computational advantages?

These definitions are equivalent.

In the equations from the paper, the variable $$v$$ points in the direction of the steepest descent, since it is computed as a negative derivative of the optimized function ($$-\eta \nabla f(\theta)$$). To update the weight vector in that direction we need to add it.

In the equations from Ruder, the variable $$v$$ points in the direction of the steepest increase, since it is computed as a positive ($$\eta \nabla f(\theta)$$). To update the vector in the opposite direction, we need to subtract it.

• I see makes sense. I was a bit confused cause of unwinding the final update to $\theta_{t}$ which turns out to be $\theta_{t+1}=\theta_{t}+\mu v_{t}-\eta \nabla f\left(\theta_{t}\right)$ as compared to $\theta_{t+1}=\theta_{t}-\mu v_{t}-\eta \nabla J\left(\theta\right)$ Commented Aug 15, 2019 at 8:39
• Is it clear to you now? Commented Aug 15, 2019 at 8:51
• Yes, thank you very much. Sorry for late reply Commented Sep 11, 2019 at 6:13