# gradient descent momentum vs step size

In the gradient descent method, the learning rate (which is multiplied by the results of the gradient on each weight) identifies the size of the step (steep down) that the algorithm takes in each iteration to reduce the objective cost function. How does it differ from the parameter momentum?

Thank you

Momentum is a whole different method, that uses parameter that works as an average of previous gradients.

Precisely in Gradient Descent (let's denote learning rate by $\eta$)

$$w_{i+1} = w_i - \eta \nabla F(w)$$

Whereas in Momentum Method

$$w_{i+1} = w_i - \gamma v_i$$

Where

$$v_{i+1} = \beta v_i + (1 - \beta) \nabla F(w)$$

Note that this method has two hyperparameters, instead of one like in GD, so I can't be sure if your momentum means $\gamma$ or $\beta$. If you use some software though, it should have two parameters.

• Thank you very much for the clear explanation. I am using (stochastic gradient descent SGD) implementation by scikit-learn here scikit-learn.org/stable/modules/generated/…. I thought Momentum is another parameter of SDG (as stated in the link) but you are saying it is not Commented Feb 18, 2018 at 21:57
• Shouldn't you swap out $\gamma$ with $\eta$? They are the same thing and you are confusing OP. Commented Feb 18, 2018 at 21:59
• @generic_user I wanted to emphasise that these are two different methods. I also think this way is actually less confusing. Commented Feb 18, 2018 at 22:06
• They're really not. Momentum is just gradient descent with an extra feature. Commented Feb 18, 2018 at 22:07
• @generic_user well that's an opinion. They are different in that they update the weights with different values. Also, don't you think that this discussion is more confusing than the parameter naming issue? ;) Commented Feb 18, 2018 at 22:08

Classical gradient descent has the shape:

$$\mathbf{x}_{i+1} = \mathbf{x}_i - \gamma \cdot\nabla f(\mathbf{x}_i)$$

for a differentiable function $$f$$, some $$\gamma \in \mathbb{R}$$ (the "learning rate" you referred to) and a starting point $$\mathbf{x}_{0}$$.

With "momentum" is meant the following adaption,

\begin{align} \mathbf{x}_{i+1} &= \mathbf{x}_i - \alpha \cdot \mathbf{z}_{i+1} \\ \text{whereat}\quad \mathbf{z}_{i+1} &= \beta \cdot \mathbf{z}_{i} + \nabla f(\mathbf{x}_i) \end{align}

where $$\mathbf{z}$$ can be interpreted as a momentum.

If you write down the closed form, you get

$$\mathbf{x}_{i+1} = \mathbf{x}_{0} - \gamma \cdot \sum^i_{k=0} \nabla f(\mathbf{x}_k)$$

for the "classical" version and

$$\mathbf{x}_{i+1} = \mathbf{x}_{0} - \alpha \cdot \sum^i_{k=0} \dfrac{1-\beta^{i-k+1}}{1-\beta}\nabla f(\mathbf{x}_k)$$

for the "momentum" version.

As you can see, they are both pretty much the same (the class is called Linear First Order Methods). Momentum just puts greater emphasis on the recent gradients, whereas the classic version treats them equally (as illustrated below).