what is the correct formula of momentum for gradient descent?

Question

I have been trying to get a better understanding of momentum, but in my search for clarification I got pretty confused. The main reason is that there seem to be multiple different, non-equivalent formulations.

Lets consider the example of gradient descent of some objective $J(\theta)$ with step size $\eta$ and momentum $\mu$.The first formulation I learnt, uses a weighted sum of the last 2 gradients, i.e. $$\begin{align*} v & \gets \eta \nabla_\theta J(\theta) \\ \theta & \gets \theta - (v + \mu v_\mathrm{old}) \\ v_\mathrm{old} & \gets v \end{align*}$$ This formulation can also be found in the efficient backprop paper.

While looking around on the web, it seems to be more common to use not only the last 2, but rather the entire history of gradients, i.e. $$\begin{align*} v & \gets \mu v + \eta \nabla_\theta J(\theta)\\ \theta & \gets \theta - v \end{align*}$$
edit: Most DL frameworks seem to implement this version.

But then I also found this article where the momentum is computed as $$\begin{align*} v & \gets \mu v + \nabla_\theta J(\theta)\\ \theta & \gets \theta - \eta v\,, \end{align*}$$ which simply gives the momentum term a different interpretation and is equivalent to the above formulation with a momentum term of $\eta \mu$.
edit: Pytorch has implemented this version.

The paper of Polyak, who is quoted as the creator of momentum, is a bit above my level and further I could not find good references that could tell me where all of these formulations come from. So I am left with my questions

1. Are there any advantages/disadvantages between the first and last two versions?
2. Is there one of the last two formulations where the momentum term has a more intuitive explanation?
3. Is there any reason why all of these different versions exist?

Answer 1

After having this question in the back of my head for some years, I think I came to some sort of agreement with myself as to what the "best" approach is: $$\begin{align*} v & \gets \mu v + (1 - \mu) \nabla_\theta J(\theta)\\ \theta & \gets \theta - \eta v\,, \end{align*}.$$

The answers to my questions below might give some arguments as to why I will probably stick to the formulation above:

1. As already noted in the comments, the second version of the exponential average has the (what I would call) advantage that if the learning rate is decreased, we do not simply keep going in the direction we were going thus far (since we make smaller changes to $v$), but effectively make smaller updates.
2. In none of the formulations, $\mu$ is really intuitive (to me), but in the last version, the learning rate is clearly the step size of the update. In the second formulation, the learning rate acts as some sort of dampening factor on the gradients rather than a step size. Therefore, I would argue that again the last formulation is more intuitive. By using a more traditional formulation of the exponential moving average (the formula above), $\mu$ is the degree of weighting decrease from the exponential moving average.
3. The simple moving average seems to be little more than some heritage from the early days of momentum. Although it does allow for changing the direction more swiftly, I do not think that there is any compelling reason to prefer it over the exponential moving average. As to where the differences between the two versions listed above come from, we would probably need to ask some mathematical historian. I would be inclined to say that there is no particular reason.

---

Addendum

Note that for $t \geq 1$ $$\begin{cases} v_0 = 0 \\ v_t = \mu v_{t-1} + (1 - \mu) \nabla_\theta J(\theta_t) \end{cases} = (1 - \mu) \begin{cases} v_0 = \nabla_\theta J(\theta_1) \\ v_t = \mu v_{t-1} + (1 - \mu) \nabla_\theta J(\theta_t) \end{cases}$$ and

$$\begin{cases} v_0 = \nabla_\theta J(\theta_1) \\ v_t = \mu v_{t-1} + (1 - \mu) \nabla_\theta J(\theta_t) \end{cases} = \begin{cases} v_0 = 0 \\ v_t = \mu v_{t-1} + (1 - \mu) \nabla_\theta J(\theta_t) \end{cases}.$$ As a result, the last exponential moving average in the question can be interpreted as a the convex combination formulation from the answer that uses the first gradient at prior instead of zero. This also shows that taking the prior at zero corresponds to scaling the learning rate with $(1 - \mu)$.