Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of initialization and momentum in deep learning, (Sutskever et al., 2013) which has a really nice explanation on it. But I realized that the algorithm was different compared to that on most websites such as http://ruder.io/optimizing-gradient-descent/index.html#momentum

$$ \begin{aligned} v_{t+1} &=\mu v_{t}-\eta \nabla f\left(\theta_{t}\right) \\ \theta_{t+1} &=\theta_{t}+v_{t+1} \end{aligned} $$

And here's the one used widely, $$ \begin{aligned} v_{t+1} &=\mu v_{t}+\eta \nabla_{\theta} J(\theta) \\ \theta_{t+1} &=\theta_{t}-v_{t+1} \end{aligned} $$

Clearly, they are different as the $\mu v_{t}$ term is added in the first update while being substracted in the second. Similarily for NAG updates, the paper implements

$$ \begin{aligned} v_{t+1} &=\mu v_{t}-\eta \nabla f\left(\theta_{t}+\mu v_{t}\right) \\ \theta_{t+1} &=\theta_{t}+v_{t+1} \end{aligned} $$

While the commonly used version is,

$$ \begin{aligned} v_{t+1} &=\mu v_{t}+\eta \nabla_{\theta} J\left(\theta-\eta v_{t}\right) \\ \theta_{t+1} &=\theta_{t}-v_{t+1} \end{aligned} $$

I have tried the implemented the second algorithm in the Coursera course and it works nicely but surely the one in the paper has also been implemented. So, are both correct and or the newer version has some computational advantages?


These definitions are equivalent.

In the equations from the paper, the variable $v$ points in the direction of the steepest descent, since it is computed as a negative derivative of the optimized function ($-\eta \nabla f(\theta)$). To update the weight vector in that direction we need to add it.

In the equations from Ruder, the variable $v$ points in the direction of the steepest increase, since it is computed as a positive ($\eta \nabla f(\theta)$). To update the vector in the opposite direction, we need to subtract it.

  • $\begingroup$ I see makes sense. I was a bit confused cause of unwinding the final update to $\theta_{t}$ which turns out to be $\theta_{t+1}=\theta_{t}+\mu v_{t}-\eta \nabla f\left(\theta_{t}\right)$ as compared to $\theta_{t+1}=\theta_{t}-\mu v_{t}-\eta \nabla J\left(\theta\right)$ $\endgroup$ – numbpy Aug 15 '19 at 8:39
  • $\begingroup$ Is it clear to you now? $\endgroup$ – Jan Kukacka Aug 15 '19 at 8:51
  • $\begingroup$ Yes, thank you very much. Sorry for late reply $\endgroup$ – numbpy Sep 11 '19 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.