2
$\begingroup$

I have been trying to get a better understanding of momentum, but in my search for clarification I got pretty confused. The main reason is that there seem to be multiple different, non-equivalent formulations.

Lets consider the example of gradient descent of some objective $J(\theta)$ with step size $\eta$ and momentum $\mu$.The first formulation I learnt, uses a weighted sum of the last 2 gradients, i.e. $$\begin{align*} v & \gets \eta \nabla_\theta J(\theta) \\ \theta & \gets \theta - (v + \mu v_\mathrm{old}) \\ v_\mathrm{old} & \gets v \end{align*}$$ This formulation can also be found in the efficient backprop paper.

While looking around on the web, it seems to be more common to use not only the last 2, but rather the entire history of gradients, i.e. $$\begin{align*} v & \gets \mu v + \eta \nabla_\theta J(\theta)\\ \theta & \gets \theta - v \end{align*}$$
edit: Most DL frameworks seem to implement this version.

But then I also found this article where the momentum is computed as $$\begin{align*} v & \gets \mu v + \nabla_\theta J(\theta)\\ \theta & \gets \theta - \eta v\,, \end{align*}$$ which simply gives the momentum term a different interpretation and is equivalent to the above formulation with a momentum term of $\eta \mu$.
edit: Pytorch has implemented this version.

The paper of Polyak, who is quoted as the creator of momentum, is a bit above my level and further I could not find good references that could tell me where all of these formulations come from. So I am left with my questions

  1. Are there any advantages/disadvantages between the first and last two versions?
  2. Is there one of the last two formulations where the momentum term has a more intuitive explanation?
  3. Is there any reason why all of these different versions exist?
$\endgroup$
  • $\begingroup$ The one that you are calling "entire history of the gradients" is commonly referred to as an exponential moving average. $\endgroup$ – Aaron Aug 26 '17 at 19:56
  • $\begingroup$ @Aaron Thanks, didn't know that. Am I correct that what I called "only the last 2 gradients" is a simple moving average then? $\endgroup$ – Mr Tsjolder Aug 26 '17 at 21:02
  • $\begingroup$ Yes. I think exponential moving average is much more common. $\endgroup$ – Aaron Aug 26 '17 at 22:04
  • $\begingroup$ IIRC the averaging techniques have varying amounts of compensation for bias, etc. These slides give a decent overview with pictures $\endgroup$ – combo Aug 30 '17 at 18:33
  • $\begingroup$ I noticed that the last two versions behave differently when using some learning rate decay. Concretely, the exponential averaging (the most popular one) puts more weight on the direction from the larger learning rate than on the new update directions. $\endgroup$ – Mr Tsjolder Feb 8 '19 at 11:21
0
$\begingroup$

After having this question in the back of my head for some years, I think I came to some sort of agreement with myself as to what the "best" approach is: $$\begin{align*} v & \gets \mu v + (1 - \mu) \nabla_\theta J(\theta)\\ \theta & \gets \theta - \eta v\,, \end{align*}.$$

The answers to my questions below might give some arguments as to why I will probably stick to the formulation above:

  1. As already noted in the comments, the second version of the exponential average has the (what I would call) advantage that if the learning rate is decreased, we do not simply keep going in the direction we were going thus far (since we make smaller changes to $v$), but effectively make smaller updates.
  2. In none of the formulations, $\mu$ is really intuitive (to me), but in the last version, the learning rate is clearly the step size of the update. In the second formulation, the learning rate acts as some sort of dampening factor on the gradients rather than a step size. Therefore, I would argue that again the last formulation is more intuitive. By using a more traditional formulation of the exponential moving average (the formula above), $\mu$ is the degree of weighting decrease from the exponential moving average.
  3. The simple moving average seems to be little more than some heritage from the early days of momentum. Although it does allow for changing the direction more swiftly, I do not think that there is any compelling reason to prefer it over the exponential moving average. As to where the differences between the two versions listed above come from, we would probably need to ask some mathematical historian. I would be inclined to say that there is no particular reason.
| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ It seems worth pointing out that the version that you mention here is a convex combination of the gradient and the previous update direction when $0 \le \mu \le 1$. en.wikipedia.org/wiki/Convex_combination $\endgroup$ – Sycorax Nov 15 '19 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.