I have been trying to get a better understanding of momentum, but in my search for clarification I got pretty confused. The main reason is that there seem to be multiple different, non-equivalent formulations.

Lets consider the example of gradient descent of some objective $J(\theta)$ with step size $\eta$ and momentum $\mu$.The first formulation I learnt, uses a weighted sum of the last 2 gradients, i.e. $$\begin{align*} v & \gets \eta \nabla_\theta J(\theta) \\ \theta & \gets \theta - (v + \mu v_\mathrm{old}) \\ v_\mathrm{old} & \gets v \end{align*}$$ This formulation can also be found in the efficient backprop paper.

While looking around on the web, it seems to be more common to use not only the last 2, but rather the entire history of gradients, i.e. $$\begin{align*} v & \gets \mu v + \eta \nabla_\theta J(\theta)\\ \theta & \gets \theta - v \end{align*}$$
edit: Most DL frameworks seem to implement this version.

But then I also found this article where the momentum is computed as $$\begin{align*} v & \gets \mu v + \nabla_\theta J(\theta)\\ \theta & \gets \theta - \eta v\,, \end{align*}$$ which simply gives the momentum term a different interpretation and is equivalent to the above formulation with a momentum term of $\eta \mu$.
edit: Pytorch has implemented this version.

The paper of Polyak, who is quoted as the creator of momentum, is a bit above my level and further I could not find good references that could tell me where all of these formulations come from. So I am left with my questions

  1. Are there any advantages/disadvantages between the first and last two versions?
  2. Is there one of the last two formulations where the momentum term has a more intuitive explanation?
  3. Is there any reason why all of these different versions exist?
  • $\begingroup$ The one that you are calling "entire history of the gradients" is commonly referred to as an exponential moving average. $\endgroup$ – Aaron Aug 26 '17 at 19:56
  • $\begingroup$ @Aaron Thanks, didn't know that. Am I correct that what I called "only the last 2 gradients" is a simple moving average then? $\endgroup$ – Mr Tsjolder Aug 26 '17 at 21:02
  • $\begingroup$ Yes. I think exponential moving average is much more common. $\endgroup$ – Aaron Aug 26 '17 at 22:04
  • $\begingroup$ IIRC the averaging techniques have varying amounts of compensation for bias, etc. These slides give a decent overview with pictures $\endgroup$ – combo Aug 30 '17 at 18:33
  • $\begingroup$ I noticed that the last two versions behave differently when using some learning rate decay. Concretely, the exponential averaging (the most popular one) puts more weight on the direction from the larger learning rate than on the new update directions. $\endgroup$ – Mr Tsjolder Feb 8 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.